2024-05-15T12:09:16.417 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-15T12:09:16.417 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --archive /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091 --owner scheduled_rzarzynski@teuthology -- /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091/orig.config.yaml 2024-05-15T12:09:16.455 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T12:09:16.548 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091 branch: wip-osd-ec-partial-reads description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7707091' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-osd-ec-partial-reads ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-osd-ec-partial-reads sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 owner: scheduled_rzarzynski@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8148 sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-osd-ec-partial-reads suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEmLW9QW9ln5PsrW3X0qHTFkDw20rYYIOHorXizGP+J22ldFT2fYmnZS5JhVJgCj+5mKoAw8uR2ftY5DxXiwXI= smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDHUNiwRS4lgXULAUrwjuTa4tocxkAsPXngfwENURgnL63jMvpSce2NjqdKwO3nce9ZCa2Zb7XDQ1qUVJKThM/Q= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-15_11:53:37 tube: smithi user: rzarzynski verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-15T12:09:16.548 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa; will attempt to use it 2024-05-15T12:09:16.548 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks 2024-05-15T12:09:16.549 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-15T12:09:16.550 INFO:teuthology.task.internal:Checking packages... 2024-05-15T12:09:16.574 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ae2481253f9b2fa3451954e615eedb67ebdd5314' 2024-05-15T12:09:16.574 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:09:16.574 INFO:teuthology.packaging:ref: None 2024-05-15T12:09:16.574 INFO:teuthology.packaging:tag: None 2024-05-15T12:09:16.574 INFO:teuthology.packaging:branch: wip-osd-ec-partial-reads 2024-05-15T12:09:16.574 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:09:16.574 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-osd-ec-partial-reads 2024-05-15T12:09:16.717 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1882.gae248125 2024-05-15T12:09:16.718 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-15T12:09:16.725 INFO:teuthology.task.internal:no buildpackages task found 2024-05-15T12:09:16.725 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-15T12:09:16.737 INFO:teuthology.task.internal:Saving configuration 2024-05-15T12:09:16.750 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-15T12:09:16.756 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T12:09:16.780 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:03:26.027077', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEmLW9QW9ln5PsrW3X0qHTFkDw20rYYIOHorXizGP+J22ldFT2fYmnZS5JhVJgCj+5mKoAw8uR2ftY5DxXiwXI='} 2024-05-15T12:09:16.803 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:03:26.025543', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDHUNiwRS4lgXULAUrwjuTa4tocxkAsPXngfwENURgnL63jMvpSce2NjqdKwO3nce9ZCa2Zb7XDQ1qUVJKThM/Q='} 2024-05-15T12:09:16.803 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-15T12:09:16.809 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-05-15T12:09:16.809 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-05-15T12:09:16.809 INFO:teuthology.run_tasks:Running task console_log... 2024-05-15T12:09:16.905 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4df81ac160>, signals=[15]) 2024-05-15T12:09:16.905 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-15T12:09:16.917 INFO:teuthology.task.internal:Opening connections... 2024-05-15T12:09:16.917 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T12:09:16.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:09:16.996 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T12:09:16.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:09:17.072 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-15T12:09:17.079 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2024-05-15T12:09:17.097 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2024-05-15T12:09:17.097 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2024-05-15T12:09:17.155 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2024-05-15T12:09:17.155 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="9" 2024-05-15T12:09:17.155 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2024-05-15T12:09:17.155 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2024-05-15T12:09:17.155 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="9" 2024-05-15T12:09:17.156 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el9" 2024-05-15T12:09:17.156 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-15T12:09:17.156 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2024-05-15T12:09:17.156 INFO:teuthology.orchestra.run.smithi049.stdout:LOGO="fedora-logo-icon" 2024-05-15T12:09:17.156 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-15T12:09:17.156 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2024-05-15T12:09:17.156 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T12:09:17.156 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-15T12:09:17.156 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T12:09:17.157 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2024-05-15T12:09:17.289 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-05-15T12:09:17.307 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-05-15T12:09:17.308 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-05-15T12:09:17.364 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="9" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="9" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el9" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:LOGO="fedora-logo-icon" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-15T12:09:17.365 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T12:09:17.366 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-05-15T12:09:17.395 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-15T12:09:17.405 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-15T12:09:17.529 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-15T12:09:17.529 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2024-05-15T12:09:17.532 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-05-15T12:09:17.548 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-15T12:09:17.618 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-15T12:09:17.626 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2024-05-15T12:09:17.628 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-05-15T12:09:17.657 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-15T12:09:17.717 INFO:teuthology.run_tasks:Running task kernel... 2024-05-15T12:09:17.732 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-15T12:09:17.732 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-15T12:09:17.732 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T12:09:17.733 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T12:09:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:09:17.749 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-05-15T12:09:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:09:17.751 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-05-15T12:09:17.807 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-437.el9.x86_64 2024-05-15T12:09:17.807 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.14.0-437.el9.x86_64 2024-05-15T12:09:17.807 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2024-05-15T12:09:17.808 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-437.el9.x86_64 2024-05-15T12:09:17.809 INFO:teuthology.task.kernel:Running kernel on smithi049: 5.14.0-437.el9.x86_64 2024-05-15T12:09:17.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2024-05-15T12:09:20.081 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-05-15T12:09:20.343 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.1 MB 00:00 2024-05-15T12:09:22.735 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-05-15T12:09:25.831 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 4.1 MB/s | 19 MB 00:04 2024-05-15T12:09:27.052 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.2 MB 00:00 2024-05-15T12:09:29.355 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-05-15T12:09:29.932 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-15T12:09:30.742 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-05-15T12:09:31.532 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 16 kB 00:00 2024-05-15T12:09:33.217 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-15T12:09:36.006 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-05-15T12:09:38.088 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-15T12:09:38.088 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-15T12:09:38.173 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 5.14.0-446.el9 baseos 550 k 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core x86_64 5.14.0-446.el9 baseos 15 M 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules x86_64 5.14.0-446.el9 baseos 34 M 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core x86_64 5.14.0-446.el9 baseos 28 M 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:09:38.187 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:09:38.188 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:09:38.188 INFO:teuthology.orchestra.run.smithi049.stdout:Install 4 Packages 2024-05-15T12:09:38.188 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:09:38.188 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 78 M 2024-05-15T12:09:38.188 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 123 M 2024-05-15T12:09:38.188 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:09:38.408 INFO:teuthology.orchestra.run.smithi049.stdout:(1/4): kernel-5.14.0-446.el9.x86_64.rpm 3.1 MB/s | 550 kB 00:00 2024-05-15T12:09:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-05-15T12:09:38.850 INFO:teuthology.orchestra.run.smithi049.stdout:(2/4): kernel-core-5.14.0-446.el9.x86_64.rpm 25 MB/s | 15 MB 00:00 2024-05-15T12:09:39.416 INFO:teuthology.orchestra.run.smithi049.stdout:(3/4): kernel-modules-5.14.0-446.el9.x86_64.rpm 29 MB/s | 34 MB 00:01 2024-05-15T12:09:39.833 INFO:teuthology.orchestra.run.smithi049.stdout:(4/4): kernel-modules-core-5.14.0-446.el9.x86_6 20 MB/s | 28 MB 00:01 2024-05-15T12:09:39.834 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:09:39.834 INFO:teuthology.orchestra.run.smithi049.stdout:Total 47 MB/s | 78 MB 00:01 2024-05-15T12:09:40.372 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-15T12:09:40.372 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-15T12:09:40.452 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:09:40.464 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:09:40.464 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T12:09:40.464 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 5.14.0-446.el9 baseos 550 k 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 5.14.0-446.el9 baseos 15 M 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 5.14.0-446.el9 baseos 34 M 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core x86_64 5.14.0-446.el9 baseos 28 M 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout:Install 4 Packages 2024-05-15T12:09:40.465 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:09:40.466 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 78 M 2024-05-15T12:09:40.466 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 123 M 2024-05-15T12:09:40.466 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:09:40.544 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:09:40.893 INFO:teuthology.orchestra.run.smithi088.stdout:(1/4): kernel-5.14.0-446.el9.x86_64.rpm 3.4 MB/s | 550 kB 00:00 2024-05-15T12:09:40.947 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:09:40.948 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:09:41.210 INFO:teuthology.orchestra.run.smithi088.stdout:(2/4): kernel-core-5.14.0-446.el9.x86_64.rpm 33 MB/s | 15 MB 00:00 2024-05-15T12:09:41.676 INFO:teuthology.orchestra.run.smithi088.stdout:(3/4): kernel-modules-5.14.0-446.el9.x86_64.rpm 36 MB/s | 34 MB 00:00 2024-05-15T12:09:41.882 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:09:41.882 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:09:41.985 INFO:teuthology.orchestra.run.smithi088.stdout:(4/4): kernel-modules-core-5.14.0-446.el9.x86_6 26 MB/s | 28 MB 00:01 2024-05-15T12:09:41.985 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:09:41.985 INFO:teuthology.orchestra.run.smithi088.stdout:Total 51 MB/s | 78 MB 00:01 2024-05-15T12:09:42.672 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:09:43.028 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:09:43.029 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:09:43.969 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:09:43.969 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:09:44.399 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:09:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-core-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:09:45.117 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:09:46.359 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:09:46.976 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-core-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:09:47.068 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:09:47.920 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:09:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:09:49.734 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:09:49.877 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:09:54.614 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:09:54.663 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:09:58.270 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:09:59.407 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:10:28.887 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:10:28.925 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:10:29.294 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:10:29.294 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:10:29.294 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:10:29.294 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:10:29.565 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:10:29.606 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:10:29.959 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:10:29.959 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:10:29.960 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:10:29.960 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:10:30.119 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:10:30.119 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:10:30.119 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:10:30.119 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-5.14.0-446.el9.x86_64 2024-05-15T12:10:30.119 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core-5.14.0-446.el9.x86_64 2024-05-15T12:10:30.119 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-5.14.0-446.el9.x86_64 2024-05-15T12:10:30.119 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core-5.14.0-446.el9.x86_64 2024-05-15T12:10:30.119 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:10:30.119 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:10:30.459 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:10:30.539 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:10:30.539 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:10:30.539 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-446.el9.x86_64 2024-05-15T12:10:30.539 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T12:10:30.539 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T12:10:30.539 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-15T12:10:30.539 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T12:10:30.540 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:10:30.589 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:10:30.589 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:10:30.589 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2024-05-15T12:10:30.620 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2024-05-15T12:10:30.620 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2024-05-15T12:10:30.620 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.06 2024-05-15T12:10:30.620 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 68.el9 2024-05-15T12:10:30.620 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2024-05-15T12:10:30.620 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-15T12:10:30.620 INFO:teuthology.orchestra.run.smithi088.stdout:Group : Unspecified 2024-05-15T12:10:30.620 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 8264254 2024-05-15T12:10:30.620 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : builder@centos.org 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2024-05-15T12:10:30.621 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:10:30.622 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T12:10:30.623 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T12:10:30.623 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 5.14.0-446.el9.x86_64 2024-05-15T12:10:30.623 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T12:10:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:10:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:10:30.795 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:10:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-5.14.0-446.el9.x86_64 2024-05-15T12:10:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core-5.14.0-446.el9.x86_64 2024-05-15T12:10:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-5.14.0-446.el9.x86_64 2024-05-15T12:10:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core-5.14.0-446.el9.x86_64 2024-05-15T12:10:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:10:30.796 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:10:31.138 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:10:31.218 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:10:31.218 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:10:31.218 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-446.el9.x86_64 2024-05-15T12:10:31.218 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T12:10:31.219 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T12:10:31.219 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-15T12:10:31.219 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T12:10:31.219 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:10:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-05-15T12:10:31.272 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:10:31.272 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:10:31.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo rpm -qi grub2-tools 2024-05-15T12:10:31.313 INFO:teuthology.orchestra.run.smithi049.stdout:Name : grub2-tools 2024-05-15T12:10:31.313 INFO:teuthology.orchestra.run.smithi049.stdout:Epoch : 1 2024-05-15T12:10:31.313 INFO:teuthology.orchestra.run.smithi049.stdout:Version : 2.06 2024-05-15T12:10:31.313 INFO:teuthology.orchestra.run.smithi049.stdout:Release : 68.el9 2024-05-15T12:10:31.313 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: x86_64 2024-05-15T12:10:31.313 INFO:teuthology.orchestra.run.smithi049.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-15T12:10:31.313 INFO:teuthology.orchestra.run.smithi049.stdout:Group : Unspecified 2024-05-15T12:10:31.313 INFO:teuthology.orchestra.run.smithi049.stdout:Size : 8264254 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:License : GPLv3+ 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:Packager : builder@centos.org 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:Vendor : CentOS 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:Summary : Support tools for GRUB. 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:Description : 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T12:10:31.314 INFO:teuthology.orchestra.run.smithi049.stdout:hardware devices. 2024-05-15T12:10:31.315 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:10:31.315 INFO:teuthology.orchestra.run.smithi049.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T12:10:31.315 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T12:10:31.316 INFO:teuthology.task.kernel:Updating grub on smithi049 to boot 5.14.0-446.el9.x86_64 2024-05-15T12:10:31.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T12:10:31.936 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2024-05-15T12:10:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-15T12:10:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-05-15T12:10:32.724 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-05-15T12:10:32.744 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.EXEABr6YuK 2024-05-15T12:10:32.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EXEABr6YuK 2024-05-15T12:10:32.817 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.EXEABr6YuK 2024-05-15T12:10:32.952 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.EXEABr6YuK is 6KB 2024-05-15T12:10:32.971 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.EXEABr6YuK 2024-05-15T12:10:32.985 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T12:10:33.055 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-15T12:10:33.055 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-15T12:10:33.056 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-15T12:10:33.056 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64.conf 2024-05-15T12:10:33.057 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64 2024-05-15T12:10:33.213 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-05-15T12:10:33.400 INFO:teuthology.orchestra.run.smithi049.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-15T12:10:33.420 INFO:teuthology.orchestra.run.smithi049.stderr:done 2024-05-15T12:10:33.421 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-05-15T12:10:33.437 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.Y9jtKPtzsb 2024-05-15T12:10:33.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Y9jtKPtzsb 2024-05-15T12:10:33.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.Y9jtKPtzsb 2024-05-15T12:10:33.644 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.Y9jtKPtzsb is 6KB 2024-05-15T12:10:33.692 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.Y9jtKPtzsb 2024-05-15T12:10:33.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T12:10:33.779 INFO:teuthology.orchestra.run.smithi049.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-15T12:10:33.779 INFO:teuthology.orchestra.run.smithi049.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-15T12:10:33.779 INFO:teuthology.orchestra.run.smithi049.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-15T12:10:33.779 INFO:teuthology.orchestra.run.smithi049.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64.conf 2024-05-15T12:10:33.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64 2024-05-15T12:10:33.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2024-05-15T12:11:03.244 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-15T12:11:03.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T12:11:03.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:11:03.934 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-15T12:11:03.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-05-15T12:11:03.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:11:21.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-05-15T12:11:22.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-05-15T12:11:30.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T12:11:30.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:11:31.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-05-15T12:11:31.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:11:33.826 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-05-15T12:11:34.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-05-15T12:11:45.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T12:11:45.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:11:46.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-05-15T12:11:46.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:11:48.898 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-05-15T12:11:49.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-05-15T12:12:03.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T12:12:03.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:12:04.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-05-15T12:12:04.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:12:35.858 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-05-15T12:12:36.225 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T12:12:36.225 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-446.el9.x86_64"... 2024-05-15T12:12:36.225 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-05-15T12:12:36.242 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-446.el9.x86_64 2024-05-15T12:12:36.242 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-446.el9.x86_64 vs 5.14.0-446.el9.x86_64 2024-05-15T12:12:36.242 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T12:12:36.242 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T12:12:37.244 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-15T12:12:37.244 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T12:12:37.306 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-05-15T12:12:37.335 DEBUG:teuthology.parallel:result is None 2024-05-15T12:13:04.410 DEBUG:teuthology.orchestra.remote:timed out 2024-05-15T12:13:22.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-05-15T12:13:22.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:13:22.675 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-05-15T12:13:23.038 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-05-15T12:13:23.038 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-446.el9.x86_64"... 2024-05-15T12:13:23.038 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-05-15T12:13:23.057 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-446.el9.x86_64 2024-05-15T12:13:23.057 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-446.el9.x86_64 vs 5.14.0-446.el9.x86_64 2024-05-15T12:13:23.057 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T12:13:23.057 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T12:13:24.058 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-15T12:13:24.058 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T12:13:24.120 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2024-05-15T12:13:24.153 DEBUG:teuthology.parallel:result is None 2024-05-15T12:13:24.154 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-15T12:13:24.199 INFO:teuthology.task.internal:Creating test directory... 2024-05-15T12:13:24.199 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T12:13:24.202 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T12:13:24.220 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-15T12:13:24.227 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-15T12:13:24.235 INFO:teuthology.task.internal:Creating archive directory... 2024-05-15T12:13:24.235 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T12:13:24.260 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T12:13:24.308 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-15T12:13:24.315 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-15T12:13:24.315 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T12:13:24.332 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T12:13:24.370 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:13:24.375 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:13:24.382 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:13:24.387 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:13:24.388 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-15T12:13:24.395 INFO:teuthology.task.internal:Configuring sudo... 2024-05-15T12:13:24.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T12:13:24.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T12:13:24.457 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-15T12:13:24.466 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-15T12:13:24.466 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T12:13:24.496 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T12:13:24.513 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:13:24.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:13:24.646 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:13:24.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:13:24.771 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:13:24.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T12:13:24.838 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:13:24.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:13:24.928 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:13:25.012 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:13:25.053 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:13:25.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T12:13:25.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2024-05-15T12:13:25.120 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-05-15T12:13:25.151 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:13:25.197 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:13:25.611 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-15T12:13:25.623 INFO:teuthology.task.internal:Starting timer... 2024-05-15T12:13:25.624 INFO:teuthology.run_tasks:Running task pcp... 2024-05-15T12:13:25.638 INFO:teuthology.run_tasks:Running task selinux... 2024-05-15T12:13:25.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2024-05-15T12:13:25.728 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2024-05-15T12:13:25.731 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-05-15T12:13:25.775 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-05-15T12:13:25.777 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-15T12:13:25.777 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2024-05-15T12:13:25.807 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2024-05-15T12:13:25.807 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-05-15T12:13:25.851 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-05-15T12:13:25.851 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi088.front.sepia.ceph.com': 'permissive'} 2024-05-15T12:13:25.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:13:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:13:25.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:13:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:13:25.918 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-15T12:13:25.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2024-05-15T12:13:25.948 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-05-15T12:13:25.984 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-15T12:13:25.994 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-15T12:13:26.011 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-15T12:13:26.012 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com,smithi088.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-15T12:17:49.865 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com'), Remote(name='ubuntu@smithi088.front.sepia.ceph.com')] 2024-05-15T12:17:49.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-05-15T12:17:49.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:17:49.999 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-05-15T12:17:50.083 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-05-15T12:17:50.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T12:17:50.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:17:50.236 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-05-15T12:17:50.322 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-15T12:17:50.322 INFO:teuthology.run_tasks:Running task clock... 2024-05-15T12:17:50.343 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-15T12:17:50.344 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T12:17:50.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:17:50.347 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T12:17:50.347 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:17:50.376 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T12:17:50.391 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T12:17:50.394 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T12:17:50.409 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T12:17:50.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2024-05-15T12:17:50.428 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2024-05-15T12:17:50.432 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2024-05-15T12:17:50.444 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T12:17:50.445 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2024-05-15T12:17:50.458 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T12:17:50.458 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T12:17:50.471 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T12:17:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-05-15T12:17:50.527 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:17:50.528 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-05-15T12:17:50.528 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:17:50.528 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:17:50.528 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:17:50.528 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:17:50.531 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-05-15T12:17:50.533 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:17:50.533 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-05-15T12:17:50.533 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:17:50.533 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:17:50.533 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:17:50.533 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:17:50.533 INFO:teuthology.run_tasks:Running task install... 2024-05-15T12:17:50.655 DEBUG:teuthology.task.install:project ceph 2024-05-15T12:17:50.655 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}} 2024-05-15T12:17:50.655 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'} 2024-05-15T12:17:50.655 INFO:teuthology.task.install:Using flavor: default 2024-05-15T12:17:50.661 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-05-15T12:17:50.661 INFO:teuthology.task.install:extra packages: [] 2024-05-15T12:17:50.662 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'tag': None, 'wait_for_package': False} 2024-05-15T12:17:50.662 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:17:50.664 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'tag': None, 'wait_for_package': False} 2024-05-15T12:17:50.664 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:17:50.810 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/ 2024-05-15T12:17:50.810 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1882.gae248125 2024-05-15T12:17:50.882 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/ 2024-05-15T12:17:50.882 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1882.gae248125 2024-05-15T12:17:51.463 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-15T12:17:51.463 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:17:51.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-15T12:17:51.463 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-15T12:17:51.463 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:17:51.464 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-15T12:17:51.495 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-05-15T12:17:51.495 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ae2481253f9b2fa3451954e615eedb67ebdd5314/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-15T12:17:51.496 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-05-15T12:17:51.496 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ae2481253f9b2fa3451954e615eedb67ebdd5314/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-15T12:17:51.565 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-15T12:17:51.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-15T12:17:51.643 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-15T12:17:51.647 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-15T12:17:51.709 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2024-05-15T12:17:51.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2024-05-15T12:17:51.714 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2024-05-15T12:17:51.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2024-05-15T12:17:51.954 INFO:teuthology.orchestra.run.smithi088.stdout:57 files removed 2024-05-15T12:17:51.955 INFO:teuthology.orchestra.run.smithi049.stdout:57 files removed 2024-05-15T12:17:51.981 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw 2024-05-15T12:17:51.983 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw 2024-05-15T12:17:52.881 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 130 kB/s | 82 kB 00:00 2024-05-15T12:17:53.067 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 103 kB/s | 82 kB 00:00 2024-05-15T12:17:53.709 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 17 kB/s | 10 kB 00:00 2024-05-15T12:17:54.032 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 9.1 kB/s | 10 kB 00:01 2024-05-15T12:17:54.467 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-05-15T12:17:54.974 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2024-05-15T12:17:55.557 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.1 MB 00:00 2024-05-15T12:17:56.526 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.1 MB 00:02 2024-05-15T12:17:59.149 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 19 MB 00:02 2024-05-15T12:18:00.260 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 19 MB 00:02 2024-05-15T12:18:04.369 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 18 MB/s | 6.2 MB 00:00 2024-05-15T12:18:05.795 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 16 kB 00:00 2024-05-15T12:18:06.132 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.2 MB 00:03 2024-05-15T12:18:07.124 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-05-15T12:18:07.508 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 16 kB 00:00 2024-05-15T12:18:09.130 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-15T12:18:12.421 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-05-15T12:18:14.304 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-15T12:18:14.865 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:19.0.0-1882.gae248125.el9 ceph 9.8 M 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.4 M 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.1 M 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.4 M 2024-05-15T12:18:14.882 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:19.0.0-1882.gae248125.el9 ceph 20 M 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:19.0.0-1882.gae248125.el9 ceph 25 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 744 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:19.0.0-1882.gae248125.el9 ceph 501 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.1 M 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-15T12:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:19.0.0-1882.gae248125.el9 ceph 45 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:19.0.0-1882.gae248125.el9 ceph 131 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:19.0.0-1882.gae248125.el9 ceph 162 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:19.0.0-1882.gae248125.el9 ceph 321 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:19.0.0-1882.gae248125.el9 ceph 298 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:19.0.0-1882.gae248125.el9 ceph 99 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout:Installing weak dependencies: 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:18:14.884 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:18:14.885 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:18:14.885 INFO:teuthology.orchestra.run.smithi049.stdout:Install 30 Packages 2024-05-15T12:18:14.885 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 2 Packages 2024-05-15T12:18:14.885 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:18:14.885 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 58 M 2024-05-15T12:18:14.885 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:18:16.637 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:18:16.646 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:18:16.646 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:19.0.0-1882.gae248125.el9 ceph 9.8 M 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.4 M 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.1 M 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.4 M 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:19.0.0-1882.gae248125.el9 ceph 20 M 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:19.0.0-1882.gae248125.el9 ceph 25 k 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-15T12:18:16.647 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 744 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:19.0.0-1882.gae248125.el9 ceph 501 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.1 M 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:19.0.0-1882.gae248125.el9 ceph 45 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:19.0.0-1882.gae248125.el9 ceph 131 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:19.0.0-1882.gae248125.el9 ceph 162 k 2024-05-15T12:18:16.648 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:19.0.0-1882.gae248125.el9 ceph 321 k 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:19.0.0-1882.gae248125.el9 ceph 298 k 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:19.0.0-1882.gae248125.el9 ceph 99 k 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout:Installing weak dependencies: 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout:Install 30 Packages 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 2 Packages 2024-05-15T12:18:16.649 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:18:16.650 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 58 M 2024-05-15T12:18:16.650 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:18:18.422 INFO:teuthology.orchestra.run.smithi049.stdout:(1/32): ceph-base-19.0.0-1882.gae248125.el9.x86 1.8 MB/s | 5.4 MB 00:03 2024-05-15T12:18:18.589 INFO:teuthology.orchestra.run.smithi049.stdout:(2/32): ceph-radosgw-19.0.0-1882.gae248125.el9. 3.1 MB/s | 9.8 MB 00:03 2024-05-15T12:18:18.606 INFO:teuthology.orchestra.run.smithi049.stdout:(3/32): ceph-selinux-19.0.0-1882.gae248125.el9. 137 kB/s | 25 kB 00:00 2024-05-15T12:18:18.840 INFO:teuthology.orchestra.run.smithi049.stdout:(4/32): libcephfs2-19.0.0-1882.gae248125.el9.x8 2.9 MB/s | 744 kB 00:00 2024-05-15T12:18:18.865 INFO:teuthology.orchestra.run.smithi049.stdout:(5/32): libradosstriper1-19.0.0-1882.gae248125. 1.9 MB/s | 501 kB 00:00 2024-05-15T12:18:18.957 INFO:teuthology.orchestra.run.smithi049.stdout:(6/32): python3-ceph-argparse-19.0.0-1882.gae24 492 kB/s | 45 kB 00:00 2024-05-15T12:18:19.065 INFO:teuthology.orchestra.run.smithi049.stdout:(7/32): python3-ceph-common-19.0.0-1882.gae2481 1.2 MB/s | 131 kB 00:00 2024-05-15T12:18:19.516 INFO:teuthology.orchestra.run.smithi049.stdout:(8/32): python3-cephfs-19.0.0-1882.gae248125.el 359 kB/s | 162 kB 00:00 2024-05-15T12:18:19.750 INFO:teuthology.orchestra.run.smithi049.stdout:(9/32): python3-rados-19.0.0-1882.gae248125.el9 1.3 MB/s | 321 kB 00:00 2024-05-15T12:18:19.883 INFO:teuthology.orchestra.run.smithi049.stdout:(10/32): librgw2-19.0.0-1882.gae248125.el9.x86_ 4.9 MB/s | 5.1 MB 00:01 2024-05-15T12:18:19.967 INFO:teuthology.orchestra.run.smithi049.stdout:(11/32): python3-rgw-19.0.0-1882.gae248125.el9. 1.2 MB/s | 99 kB 00:00 2024-05-15T12:18:19.992 INFO:teuthology.orchestra.run.smithi049.stdout:(12/32): python3-rbd-19.0.0-1882.gae248125.el9. 1.2 MB/s | 298 kB 00:00 2024-05-15T12:18:20.134 INFO:teuthology.orchestra.run.smithi049.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 641 kB/s | 107 kB 00:00 2024-05-15T12:18:20.168 INFO:teuthology.orchestra.run.smithi049.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 913 kB/s | 160 kB 00:00 2024-05-15T12:18:20.193 INFO:teuthology.orchestra.run.smithi049.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 769 kB/s | 45 kB 00:00 2024-05-15T12:18:20.221 INFO:teuthology.orchestra.run.smithi088.stdout:(1/32): ceph-base-19.0.0-1882.gae248125.el9.x86 1.6 MB/s | 5.4 MB 00:03 2024-05-15T12:18:20.288 INFO:teuthology.orchestra.run.smithi088.stdout:(2/32): ceph-selinux-19.0.0-1882.gae248125.el9. 379 kB/s | 25 kB 00:00 2024-05-15T12:18:20.302 INFO:teuthology.orchestra.run.smithi049.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-05-15T12:18:20.352 INFO:teuthology.orchestra.run.smithi049.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-05-15T12:18:20.377 INFO:teuthology.orchestra.run.smithi049.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 562 kB/s | 42 kB 00:00 2024-05-15T12:18:20.428 INFO:teuthology.orchestra.run.smithi049.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 720 kB/s | 36 kB 00:00 2024-05-15T12:18:20.453 INFO:teuthology.orchestra.run.smithi049.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-05-15T12:18:20.470 INFO:teuthology.orchestra.run.smithi049.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-05-15T12:18:20.596 INFO:teuthology.orchestra.run.smithi088.stdout:(3/32): libcephfs2-19.0.0-1882.gae248125.el9.x8 2.4 MB/s | 744 kB 00:00 2024-05-15T12:18:20.805 INFO:teuthology.orchestra.run.smithi088.stdout:(4/32): libradosstriper1-19.0.0-1882.gae248125. 2.4 MB/s | 501 kB 00:00 2024-05-15T12:18:21.062 INFO:teuthology.orchestra.run.smithi049.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 507 kB/s | 309 kB 00:00 2024-05-15T12:18:21.162 INFO:teuthology.orchestra.run.smithi049.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-05-15T12:18:21.271 INFO:teuthology.orchestra.run.smithi049.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 449 kB/s | 49 kB 00:00 2024-05-15T12:18:21.379 INFO:teuthology.orchestra.run.smithi049.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-05-15T12:18:21.438 INFO:teuthology.orchestra.run.smithi049.stdout:(26/32): libarrow-9.0.0-11.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-05-15T12:18:21.538 INFO:teuthology.orchestra.run.smithi049.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-05-15T12:18:21.613 INFO:teuthology.orchestra.run.smithi049.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.5 MB/s | 839 kB 00:00 2024-05-15T12:18:21.672 INFO:teuthology.orchestra.run.smithi049.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-15T12:18:21.847 INFO:teuthology.orchestra.run.smithi088.stdout:(5/32): ceph-radosgw-19.0.0-1882.gae248125.el9. 2.0 MB/s | 9.8 MB 00:04 2024-05-15T12:18:21.972 INFO:teuthology.orchestra.run.smithi088.stdout:(6/32): python3-ceph-argparse-19.0.0-1882.gae24 361 kB/s | 45 kB 00:00 2024-05-15T12:18:22.081 INFO:teuthology.orchestra.run.smithi088.stdout:(7/32): python3-ceph-common-19.0.0-1882.gae2481 1.2 MB/s | 131 kB 00:00 2024-05-15T12:18:22.172 INFO:teuthology.orchestra.run.smithi088.stdout:(8/32): python3-cephfs-19.0.0-1882.gae248125.el 1.7 MB/s | 162 kB 00:00 2024-05-15T12:18:22.348 INFO:teuthology.orchestra.run.smithi088.stdout:(9/32): python3-rados-19.0.0-1882.gae248125.el9 1.8 MB/s | 321 kB 00:00 2024-05-15T12:18:22.523 INFO:teuthology.orchestra.run.smithi088.stdout:(10/32): python3-rbd-19.0.0-1882.gae248125.el9. 1.7 MB/s | 298 kB 00:00 2024-05-15T12:18:22.632 INFO:teuthology.orchestra.run.smithi088.stdout:(11/32): python3-rgw-19.0.0-1882.gae248125.el9. 918 kB/s | 99 kB 00:00 2024-05-15T12:18:22.714 INFO:teuthology.orchestra.run.smithi049.stdout:(30/32): librbd1-19.0.0-1882.gae248125.el9.x86_ 3.0 MB/s | 3.1 MB 00:01 2024-05-15T12:18:22.757 INFO:teuthology.orchestra.run.smithi088.stdout:(12/32): librgw2-19.0.0-1882.gae248125.el9.x86_ 2.6 MB/s | 5.1 MB 00:01 2024-05-15T12:18:22.966 INFO:teuthology.orchestra.run.smithi088.stdout:(13/32): ceph-common-19.0.0-1882.gae248125.el9. 3.3 MB/s | 20 MB 00:06 2024-05-15T12:18:22.991 INFO:teuthology.orchestra.run.smithi088.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 298 kB/s | 107 kB 00:00 2024-05-15T12:18:23.016 INFO:teuthology.orchestra.run.smithi088.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 908 kB/s | 45 kB 00:00 2024-05-15T12:18:23.089 INFO:teuthology.orchestra.run.smithi049.stdout:(31/32): ceph-common-19.0.0-1882.gae248125.el9. 2.6 MB/s | 20 MB 00:07 2024-05-15T12:18:23.091 INFO:teuthology.orchestra.run.smithi088.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 479 kB/s | 160 kB 00:00 2024-05-15T12:18:23.117 INFO:teuthology.orchestra.run.smithi088.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-05-15T12:18:23.142 INFO:teuthology.orchestra.run.smithi088.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-05-15T12:18:23.184 INFO:teuthology.orchestra.run.smithi088.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-05-15T12:18:23.209 INFO:teuthology.orchestra.run.smithi088.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 536 kB/s | 36 kB 00:00 2024-05-15T12:18:23.214 INFO:teuthology.orchestra.run.smithi049.stdout:(32/32): librados2-19.0.0-1882.gae248125.el9.x8 2.1 MB/s | 3.4 MB 00:01 2024-05-15T12:18:23.220 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:18:23.220 INFO:teuthology.orchestra.run.smithi049.stdout:Total 7.0 MB/s | 58 MB 00:08 2024-05-15T12:18:23.234 INFO:teuthology.orchestra.run.smithi088.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-05-15T12:18:23.268 INFO:teuthology.orchestra.run.smithi088.stdout:(22/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2024-05-15T12:18:23.333 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:18:23.374 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:18:23.375 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:18:23.519 INFO:teuthology.orchestra.run.smithi088.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 104 kB/s | 26 kB 00:00 2024-05-15T12:18:23.710 INFO:teuthology.orchestra.run.smithi088.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-05-15T12:18:23.735 INFO:teuthology.orchestra.run.smithi088.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 587 kB/s | 309 kB 00:00 2024-05-15T12:18:23.903 INFO:teuthology.orchestra.run.smithi088.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 351 kB/s | 67 kB 00:00 2024-05-15T12:18:23.997 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:18:23.998 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:18:24.061 INFO:teuthology.orchestra.run.smithi088.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.5 MB/s | 839 kB 00:00 2024-05-15T12:18:24.112 INFO:teuthology.orchestra.run.smithi088.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 919 kB/s | 192 kB 00:00 2024-05-15T12:18:24.245 INFO:teuthology.orchestra.run.smithi088.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-05-15T12:18:24.445 INFO:teuthology.orchestra.run.smithi088.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.1 MB/s | 1.6 MB 00:00 2024-05-15T12:18:24.946 INFO:teuthology.orchestra.run.smithi088.stdout:(31/32): librbd1-19.0.0-1882.gae248125.el9.x86_ 4.5 MB/s | 3.1 MB 00:00 2024-05-15T12:18:25.004 INFO:teuthology.orchestra.run.smithi088.stdout:(32/32): librados2-19.0.0-1882.gae248125.el9.x8 3.8 MB/s | 3.4 MB 00:00 2024-05-15T12:18:25.006 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:18:25.006 INFO:teuthology.orchestra.run.smithi088.stdout:Total 6.9 MB/s | 58 MB 00:08 2024-05-15T12:18:25.109 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:18:25.146 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:18:25.147 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:18:25.421 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:18:25.510 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-15T12:18:25.545 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-15T12:18:25.570 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-15T12:18:25.593 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-15T12:18:25.620 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-15T12:18:25.754 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:18:25.754 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:18:25.813 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-15T12:18:25.874 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-15T12:18:26.264 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-15T12:18:26.322 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:18:26.398 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:18:26.487 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 8/34 2024-05-15T12:18:26.523 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 9/34 2024-05-15T12:18:26.570 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 9/34 2024-05-15T12:18:26.607 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-15T12:18:26.652 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:19.0.0-1882.gae248125.el9. 11/34 2024-05-15T12:18:26.721 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 12/34 2024-05-15T12:18:26.742 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 13/34 2024-05-15T12:18:26.798 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 13/34 2024-05-15T12:18:26.833 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-15T12:18:26.961 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-15T12:18:26.994 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-15T12:18:27.045 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-15T12:18:27.074 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-15T12:18:27.103 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-15T12:18:27.135 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-15T12:18:27.164 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:18:27.278 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-15T12:18:27.309 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-15T12:18:27.342 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-15T12:18:27.366 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-15T12:18:27.389 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-15T12:18:27.466 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-15T12:18:27.507 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:19.0.0-1882.gae248125.el9.x86_64 22/34 2024-05-15T12:18:27.582 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:19.0.0-1882.gae248125.el9.x86_64 22/34 2024-05-15T12:18:27.583 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-15T12:18:27.612 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 23/34 2024-05-15T12:18:27.648 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-15T12:18:27.648 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-15T12:18:28.037 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-15T12:18:28.094 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:18:28.128 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-15T12:18:28.159 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:19.0.0-1882.gae248125.el9.x86_64 26/34 2024-05-15T12:18:28.186 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:18:28.215 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:19.0.0-1882.gae248125.el9.x86_64 26/34 2024-05-15T12:18:28.272 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 8/34 2024-05-15T12:18:28.295 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 27/34 2024-05-15T12:18:28.303 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 9/34 2024-05-15T12:18:28.329 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:19.0.0-1882.gae248125.el9.x8 28/34 2024-05-15T12:18:28.366 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 9/34 2024-05-15T12:18:28.395 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-15T12:18:28.439 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:19.0.0-1882.gae248125.el9. 11/34 2024-05-15T12:18:28.506 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 12/34 2024-05-15T12:18:28.531 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 13/34 2024-05-15T12:18:28.605 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 13/34 2024-05-15T12:18:28.638 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-15T12:18:28.754 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-15T12:18:28.815 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-15T12:18:28.856 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-15T12:18:28.887 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-15T12:18:28.916 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-15T12:18:28.949 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-15T12:18:29.261 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-15T12:18:29.312 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:19.0.0-1882.gae248125.el9.x86_64 22/34 2024-05-15T12:18:29.402 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:19.0.0-1882.gae248125.el9.x86_64 22/34 2024-05-15T12:18:29.434 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 23/34 2024-05-15T12:18:29.478 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-15T12:18:29.935 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-15T12:18:29.956 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:19.0.0-1882.gae248125.el9.x86_64 26/34 2024-05-15T12:18:30.033 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:19.0.0-1882.gae248125.el9.x86_64 26/34 2024-05-15T12:18:30.105 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 27/34 2024-05-15T12:18:30.134 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:19.0.0-1882.gae248125.el9.x8 28/34 2024-05-15T12:18:30.613 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:18:30.659 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:18:31.235 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:18:31.271 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 30/34 2024-05-15T12:18:31.363 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 30/34 2024-05-15T12:18:31.363 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-15T12:18:31.363 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-15T12:18:31.363 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:18:31.389 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:18:32.333 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:18:32.564 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:18:33.118 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 29/34 2024-05-15T12:18:33.151 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 30/34 2024-05-15T12:18:33.243 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 30/34 2024-05-15T12:18:33.243 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-15T12:18:33.243 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-15T12:18:33.244 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:18:33.277 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:18:41.516 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:18:41.517 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2024-05-15T12:18:41.517 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2024-05-15T12:18:41.517 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2024-05-15T12:18:41.517 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2024-05-15T12:18:41.517 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2024-05-15T12:18:41.517 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /root 2024-05-15T12:18:41.517 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2024-05-15T12:18:41.517 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:18:42.452 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 32/34 2024-05-15T12:18:42.478 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 32/34 2024-05-15T12:18:42.479 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:18:42.479 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-15T12:18:42.479 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-15T12:18:42.479 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-15T12:18:42.479 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:18:42.481 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-15T12:18:42.537 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:18:42.537 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2024-05-15T12:18:42.537 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2024-05-15T12:18:42.537 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2024-05-15T12:18:42.537 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2024-05-15T12:18:42.537 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2024-05-15T12:18:42.537 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /root 2024-05-15T12:18:42.538 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2024-05-15T12:18:42.538 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:18:42.562 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-15T12:18:42.563 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:18:43.469 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 32/34 2024-05-15T12:18:43.494 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 32/34 2024-05-15T12:18:43.494 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:18:43.494 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-15T12:18:43.494 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-15T12:18:43.494 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-15T12:18:43.495 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:18:43.496 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-15T12:18:43.589 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-15T12:18:43.589 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:18:43.683 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:18:43.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 1/34 2024-05-15T12:18:43.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2/34 2024-05-15T12:18:43.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 3/34 2024-05-15T12:18:43.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 4/34 2024-05-15T12:18:43.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 5/34 2024-05-15T12:18:43.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 6/34 2024-05-15T12:18:43.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:18:43.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1882.gae248125.el9. 8/34 2024-05-15T12:18:43.683 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:19.0.0-1882.gae248125.el9.x8 9/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 10/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 11/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 12/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 13/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-15T12:18:43.684 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-15T12:18:43.685 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-15T12:18:43.685 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-15T12:18:43.685 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-15T12:18:43.685 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-15T12:18:43.685 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-15T12:18:43.685 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-15T12:18:43.685 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-15T12:18:43.685 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:18:43.685 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-15T12:18:43.686 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:19.0.0-1882.gae248125.el9.x86_64 33/34 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-15T12:18:44.454 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: re2-1:20211101-3.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:18:44.455 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:18:44.694 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:18:44.694 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 1/34 2024-05-15T12:18:44.694 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2/34 2024-05-15T12:18:44.694 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 3/34 2024-05-15T12:18:44.694 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 4/34 2024-05-15T12:18:44.694 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 5/34 2024-05-15T12:18:44.694 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_6 6/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:19.0.0-1882.gae248125.el9.x86_64 7/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1882.gae248125.el9. 8/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:19.0.0-1882.gae248125.el9.x8 9/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 10/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 11/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 12/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 13/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-15T12:18:44.695 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:19.0.0-1882.gae248125.el9.x86_64 31/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-15T12:18:44.696 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:19.0.0-1882.gae248125.el9.x86_64 33/34 2024-05-15T12:18:44.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test 2024-05-15T12:18:45.431 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:33 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:18:45.507 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-15T12:18:45.508 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-15T12:18:45.509 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: re2-1:20211101-3.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-15T12:18:45.510 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:18:45.511 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:18:45.558 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:18:45.559 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:18:45.559 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-05-15T12:18:45.559 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:18:45.559 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:18:45.559 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:19.0.0-1882.gae248125.el9 ceph 47 M 2024-05-15T12:18:45.559 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:18:45.559 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:19.0.0-1882.gae248125.el9 ceph 163 k 2024-05-15T12:18:45.559 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-15T12:18:45.559 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-15T12:18:45.559 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:18:45.560 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:18:45.560 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:18:45.560 INFO:teuthology.orchestra.run.smithi049.stdout:Install 4 Packages 2024-05-15T12:18:45.560 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:18:45.560 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 48 M 2024-05-15T12:18:45.560 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 204 M 2024-05-15T12:18:45.561 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:18:45.910 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-test 2024-05-15T12:18:46.101 INFO:teuthology.orchestra.run.smithi049.stdout:(1/4): libcephsqlite-19.0.0-1882.gae248125.el9. 351 kB/s | 163 kB 00:00 2024-05-15T12:18:46.134 INFO:teuthology.orchestra.run.smithi049.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 611 kB/s | 305 kB 00:00 2024-05-15T12:18:46.426 INFO:teuthology.orchestra.run.smithi049.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 196 kB/s | 64 kB 00:00 2024-05-15T12:18:46.487 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:32 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:18:46.608 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:18:46.609 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:18:46.609 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T12:18:46.609 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:18:46.609 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:18:46.609 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:19.0.0-1882.gae248125.el9 ceph 47 M 2024-05-15T12:18:46.609 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:18:46.610 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:19.0.0-1882.gae248125.el9 ceph 163 k 2024-05-15T12:18:46.610 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-15T12:18:46.610 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-15T12:18:46.610 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:18:46.610 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:18:46.610 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:18:46.610 INFO:teuthology.orchestra.run.smithi088.stdout:Install 4 Packages 2024-05-15T12:18:46.610 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:18:46.611 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 48 M 2024-05-15T12:18:46.611 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 204 M 2024-05-15T12:18:46.611 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:18:46.835 INFO:teuthology.orchestra.run.smithi088.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 305 kB 00:00 2024-05-15T12:18:46.885 INFO:teuthology.orchestra.run.smithi088.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-05-15T12:18:47.178 INFO:teuthology.orchestra.run.smithi088.stdout:(3/4): libcephsqlite-19.0.0-1882.gae248125.el9. 309 kB/s | 163 kB 00:00 2024-05-15T12:18:56.876 INFO:teuthology.orchestra.run.smithi088.stdout:(4/4): ceph-test-19.0.0-1882.gae248125.el9.x86_ 4.6 MB/s | 47 MB 00:10 2024-05-15T12:18:56.878 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:18:56.878 INFO:teuthology.orchestra.run.smithi088.stdout:Total 4.7 MB/s | 48 MB 00:10 2024-05-15T12:18:56.907 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:18:56.928 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:18:56.928 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:18:57.290 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:18:57.290 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:18:57.810 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:18:57.918 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-15T12:18:57.958 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-15T12:18:57.986 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 3/4 2024-05-15T12:19:02.539 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 3/4 2024-05-15T12:19:02.617 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 4/4 2024-05-15T12:19:02.684 INFO:teuthology.orchestra.run.smithi049.stdout:(4/4): ceph-test-19.0.0-1882.gae248125.el9.x86_ 2.8 MB/s | 47 MB 00:17 2024-05-15T12:19:02.685 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:02.686 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.8 MB/s | 48 MB 00:17 2024-05-15T12:19:02.713 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:19:02.735 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:19:02.735 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:19:03.117 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:19:03.117 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:19:03.655 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:19:03.744 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-15T12:19:03.791 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-15T12:19:03.818 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 3/4 2024-05-15T12:19:04.879 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 4/4 2024-05-15T12:19:04.879 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 1/4 2024-05-15T12:19:04.879 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 2/4 2024-05-15T12:19:04.880 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-15T12:19:05.180 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-15T12:19:05.180 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:05.180 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:19:05.180 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:05.180 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:05.180 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-15T12:19:05.180 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-15T12:19:05.180 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:05.180 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:19:05.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph 2024-05-15T12:19:06.055 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:52 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:19:06.179 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:19.0.0-1882.gae248125.el9 ceph 6.6 k 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:19.0.0-1882.gae248125.el9 ceph 2.1 M 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.5 M 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 248 k 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:19.0.0-1882.gae248125.el9 ceph 4.7 M 2024-05-15T12:19:06.182 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:19.0.0-1882.gae248125.el9 ceph 17 M 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-15T12:19:06.183 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-15T12:19:06.184 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-15T12:19:06.185 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-15T12:19:06.185 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-15T12:19:06.185 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-15T12:19:06.185 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:06.185 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:19:06.185 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:06.185 INFO:teuthology.orchestra.run.smithi088.stdout:Install 36 Packages 2024-05-15T12:19:06.185 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:06.186 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 30 M 2024-05-15T12:19:06.186 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 107 M 2024-05-15T12:19:06.186 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:19:07.159 INFO:teuthology.orchestra.run.smithi088.stdout:(1/36): ceph-19.0.0-1882.gae248125.el9.x86_64.r 32 kB/s | 6.6 kB 00:00 2024-05-15T12:19:07.293 INFO:teuthology.orchestra.run.smithi088.stdout:(2/36): ceph-mgr-19.0.0-1882.gae248125.el9.x86_ 4.4 MB/s | 1.5 MB 00:00 2024-05-15T12:19:07.718 INFO:teuthology.orchestra.run.smithi088.stdout:(3/36): ceph-mds-19.0.0-1882.gae248125.el9.x86_ 2.8 MB/s | 2.1 MB 00:00 2024-05-15T12:19:07.826 INFO:teuthology.orchestra.run.smithi088.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1882.gae24 2.2 MB/s | 248 kB 00:00 2024-05-15T12:19:08.202 INFO:teuthology.orchestra.run.smithi088.stdout:(5/36): ceph-osd-19.0.0-1882.gae248125.el9.x86_ 19 MB/s | 17 MB 00:00 2024-05-15T12:19:08.477 INFO:teuthology.orchestra.run.smithi088.stdout:(6/36): ceph-mon-19.0.0-1882.gae248125.el9.x86_ 3.6 MB/s | 4.7 MB 00:01 2024-05-15T12:19:08.515 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 3/4 2024-05-15T12:19:08.603 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 4/4 2024-05-15T12:19:08.835 INFO:teuthology.orchestra.run.smithi088.stdout:(7/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 251 kB/s | 253 kB 00:01 2024-05-15T12:19:09.260 INFO:teuthology.orchestra.run.smithi088.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 140 kB/s | 106 kB 00:00 2024-05-15T12:19:09.294 INFO:teuthology.orchestra.run.smithi088.stdout:(9/36): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-05-15T12:19:09.669 INFO:teuthology.orchestra.run.smithi088.stdout:(10/36): python3-pycparser-2.20-6.el9.noarch.rp 162 kB/s | 135 kB 00:00 2024-05-15T12:19:09.736 INFO:teuthology.orchestra.run.smithi088.stdout:(11/36): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-05-15T12:19:09.803 INFO:teuthology.orchestra.run.smithi088.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-05-15T12:19:09.853 INFO:teuthology.orchestra.run.smithi088.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 697 kB/s | 35 kB 00:00 2024-05-15T12:19:09.912 INFO:teuthology.orchestra.run.smithi088.stdout:(14/36): python3-pytz-2021.1-5.el9.noarch.rpm 878 kB/s | 51 kB 00:00 2024-05-15T12:19:09.970 INFO:teuthology.orchestra.run.smithi088.stdout:(15/36): python3-toml-0.10.2-6.el9.noarch.rpm 715 kB/s | 42 kB 00:00 2024-05-15T12:19:10.029 INFO:teuthology.orchestra.run.smithi088.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 162 kB/s | 125 kB 00:00 2024-05-15T12:19:10.079 INFO:teuthology.orchestra.run.smithi088.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-05-15T12:19:10.138 INFO:teuthology.orchestra.run.smithi088.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-05-15T12:19:10.163 INFO:teuthology.orchestra.run.smithi088.stdout:(19/36): python3-urllib3-1.26.5-5.el9.noarch.rp 247 kB/s | 215 kB 00:00 2024-05-15T12:19:10.188 INFO:teuthology.orchestra.run.smithi088.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 400 kB/s | 43 kB 00:00 2024-05-15T12:19:10.245 INFO:teuthology.orchestra.run.smithi088.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-05-15T12:19:10.289 INFO:teuthology.orchestra.run.smithi088.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-05-15T12:19:10.314 INFO:teuthology.orchestra.run.smithi088.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 257 kB/s | 18 kB 00:00 2024-05-15T12:19:10.339 INFO:teuthology.orchestra.run.smithi088.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-05-15T12:19:10.365 INFO:teuthology.orchestra.run.smithi088.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-05-15T12:19:10.390 INFO:teuthology.orchestra.run.smithi088.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-05-15T12:19:10.415 INFO:teuthology.orchestra.run.smithi088.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-05-15T12:19:10.440 INFO:teuthology.orchestra.run.smithi088.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-05-15T12:19:10.476 INFO:teuthology.orchestra.run.smithi088.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 918 kB/s | 79 kB 00:00 2024-05-15T12:19:10.501 INFO:teuthology.orchestra.run.smithi088.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.1 MB/s | 272 kB 00:00 2024-05-15T12:19:10.527 INFO:teuthology.orchestra.run.smithi088.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 191 kB/s | 16 kB 00:00 2024-05-15T12:19:10.552 INFO:teuthology.orchestra.run.smithi088.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-05-15T12:19:10.577 INFO:teuthology.orchestra.run.smithi088.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-05-15T12:19:10.602 INFO:teuthology.orchestra.run.smithi088.stdout:(34/36): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-05-15T12:19:10.627 INFO:teuthology.orchestra.run.smithi088.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-05-15T12:19:10.686 INFO:teuthology.orchestra.run.smithi088.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-05-15T12:19:10.689 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:10.689 INFO:teuthology.orchestra.run.smithi088.stdout:Total 6.6 MB/s | 30 MB 00:04 2024-05-15T12:19:10.837 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:19:10.872 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 4/4 2024-05-15T12:19:10.872 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 1/4 2024-05-15T12:19:10.872 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 2/4 2024-05-15T12:19:10.872 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-15T12:19:10.876 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:19:10.877 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:19:11.180 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-15T12:19:11.180 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:11.180 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:19:11.180 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:11.180 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:11.180 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-15T12:19:11.180 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-15T12:19:11.180 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:11.180 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:19:11.302 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:19:11.302 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:19:11.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph 2024-05-15T12:19:12.026 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:00 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:19:12.156 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:19.0.0-1882.gae248125.el9 ceph 6.6 k 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:19.0.0-1882.gae248125.el9 ceph 2.1 M 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.5 M 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 248 k 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:19.0.0-1882.gae248125.el9 ceph 4.7 M 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:19.0.0-1882.gae248125.el9 ceph 17 M 2024-05-15T12:19:12.159 INFO:teuthology.orchestra.run.smithi049.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-15T12:19:12.160 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-15T12:19:12.161 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:12.162 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:19:12.162 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:12.162 INFO:teuthology.orchestra.run.smithi049.stdout:Install 36 Packages 2024-05-15T12:19:12.162 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:12.162 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 30 M 2024-05-15T12:19:12.162 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 107 M 2024-05-15T12:19:12.163 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:19:12.553 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:19:12.622 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-15T12:19:12.664 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-15T12:19:12.691 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-15T12:19:12.764 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-15T12:19:12.824 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-15T12:19:12.856 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-15T12:19:12.951 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-15T12:19:12.994 INFO:teuthology.orchestra.run.smithi049.stdout:(1/36): ceph-19.0.0-1882.gae248125.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-05-15T12:19:13.028 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-15T12:19:13.076 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-15T12:19:13.111 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-15T12:19:13.141 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-15T12:19:13.176 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-15T12:19:13.209 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-15T12:19:13.250 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-15T12:19:13.301 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-15T12:19:13.335 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-15T12:19:13.374 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-15T12:19:13.448 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-15T12:19:13.587 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-15T12:19:13.611 INFO:teuthology.orchestra.run.smithi049.stdout:(2/36): ceph-mgr-19.0.0-1882.gae248125.el9.x86_ 1.6 MB/s | 1.5 MB 00:00 2024-05-15T12:19:13.661 INFO:teuthology.orchestra.run.smithi049.stdout:(3/36): ceph-mds-19.0.0-1882.gae248125.el9.x86_ 2.1 MB/s | 2.1 MB 00:01 2024-05-15T12:19:13.674 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-15T12:19:13.727 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-15T12:19:13.765 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-15T12:19:13.786 INFO:teuthology.orchestra.run.smithi049.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1882.gae24 1.9 MB/s | 248 kB 00:00 2024-05-15T12:19:13.853 INFO:teuthology.orchestra.run.smithi049.stdout:(5/36): ceph-mon-19.0.0-1882.gae248125.el9.x86_ 5.5 MB/s | 4.7 MB 00:00 2024-05-15T12:19:13.878 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-15T12:19:13.936 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-15T12:19:13.979 INFO:teuthology.orchestra.run.smithi049.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-05-15T12:19:14.021 INFO:teuthology.orchestra.run.smithi049.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-05-15T12:19:14.063 INFO:teuthology.orchestra.run.smithi049.stdout:(8/36): python3-cryptography-36.0.1-4.el9.x86_6 5.9 MB/s | 1.2 MB 00:00 2024-05-15T12:19:14.088 INFO:teuthology.orchestra.run.smithi049.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-05-15T12:19:14.113 INFO:teuthology.orchestra.run.smithi049.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-05-15T12:19:14.124 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-15T12:19:14.139 INFO:teuthology.orchestra.run.smithi049.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-05-15T12:19:14.177 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-15T12:19:14.247 INFO:teuthology.orchestra.run.smithi049.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-05-15T12:19:14.248 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-15T12:19:14.272 INFO:teuthology.orchestra.run.smithi049.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-05-15T12:19:14.298 INFO:teuthology.orchestra.run.smithi049.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 689 kB/s | 35 kB 00:00 2024-05-15T12:19:14.323 INFO:teuthology.orchestra.run.smithi049.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-15T12:19:14.337 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-15T12:19:14.348 INFO:teuthology.orchestra.run.smithi049.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-05-15T12:19:14.377 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-15T12:19:14.450 INFO:teuthology.orchestra.run.smithi049.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-05-15T12:19:14.495 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-15T12:19:14.659 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9. 31/36 2024-05-15T12:19:14.696 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 32/36 2024-05-15T12:19:14.721 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 32/36 2024-05-15T12:19:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:19:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-15T12:19:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-15T12:19:14.721 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-15T12:19:14.722 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:14.765 INFO:teuthology.orchestra.run.smithi049.stdout:(18/36): ceph-osd-19.0.0-1882.gae248125.el9.x86 15 MB/s | 17 MB 00:01 2024-05-15T12:19:14.791 INFO:teuthology.orchestra.run.smithi049.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 127 kB/s | 43 kB 00:00 2024-05-15T12:19:14.866 INFO:teuthology.orchestra.run.smithi049.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 292 kB/s | 151 kB 00:00 2024-05-15T12:19:14.950 INFO:teuthology.orchestra.run.smithi049.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-05-15T12:19:15.033 INFO:teuthology.orchestra.run.smithi049.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-05-15T12:19:15.058 INFO:teuthology.orchestra.run.smithi049.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-05-15T12:19:15.117 INFO:teuthology.orchestra.run.smithi049.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 277 kB/s | 23 kB 00:00 2024-05-15T12:19:15.142 INFO:teuthology.orchestra.run.smithi049.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-05-15T12:19:15.167 INFO:teuthology.orchestra.run.smithi049.stdout:(26/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 428 kB/s | 172 kB 00:00 2024-05-15T12:19:15.201 INFO:teuthology.orchestra.run.smithi049.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 234 kB/s | 20 kB 00:00 2024-05-15T12:19:15.227 INFO:teuthology.orchestra.run.smithi049.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 552 kB/s | 46 kB 00:00 2024-05-15T12:19:15.260 INFO:teuthology.orchestra.run.smithi049.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 854 kB/s | 79 kB 00:00 2024-05-15T12:19:15.302 INFO:teuthology.orchestra.run.smithi049.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-05-15T12:19:15.352 INFO:teuthology.orchestra.run.smithi049.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 980 kB/s | 90 kB 00:00 2024-05-15T12:19:15.377 INFO:teuthology.orchestra.run.smithi049.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-05-15T12:19:15.403 INFO:teuthology.orchestra.run.smithi049.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-05-15T12:19:15.478 INFO:teuthology.orchestra.run.smithi049.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-05-15T12:19:15.503 INFO:teuthology.orchestra.run.smithi049.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-05-15T12:19:15.537 INFO:teuthology.orchestra.run.smithi049.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-05-15T12:19:15.542 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:15.542 INFO:teuthology.orchestra.run.smithi049.stdout:Total 8.8 MB/s | 30 MB 00:03 2024-05-15T12:19:15.692 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:19:15.736 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:19:15.736 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:19:16.209 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:19:16.210 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:19:16.223 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 33/36 2024-05-15T12:19:16.252 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 33/36 2024-05-15T12:19:16.252 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:19:16.252 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-15T12:19:16.253 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-15T12:19:16.253 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-15T12:19:16.253 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:16.698 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 34/36 2024-05-15T12:19:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 34/36 2024-05-15T12:19:16.723 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:19:16.723 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-15T12:19:16.723 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-15T12:19:16.723 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-15T12:19:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:16.932 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 35/36 2024-05-15T12:19:16.956 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 35/36 2024-05-15T12:19:16.956 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:19:16.956 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-15T12:19:16.956 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-15T12:19:16.956 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-15T12:19:16.957 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:17.020 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:19.0.0-1882.gae248125.el9.x86_64 36/36 2024-05-15T12:19:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:19:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-15T12:19:17.589 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-15T12:19:17.617 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-15T12:19:17.700 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-15T12:19:17.742 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-15T12:19:17.775 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-15T12:19:17.871 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-15T12:19:17.947 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-15T12:19:17.994 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-15T12:19:18.029 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-15T12:19:18.058 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-15T12:19:18.093 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-15T12:19:18.127 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-15T12:19:18.193 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-15T12:19:18.244 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-15T12:19:18.279 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-15T12:19:18.317 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-15T12:19:18.394 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-15T12:19:18.537 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-15T12:19:18.627 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-15T12:19:18.688 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-15T12:19:18.726 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-15T12:19:18.782 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-2:19.0.0-1882.gae248125.el9.x86_64 36/36 2024-05-15T12:19:18.782 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:19.0.0-1882.gae248125.el9.x86_64 1/36 2024-05-15T12:19:18.782 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 2/36 2024-05-15T12:19:18.782 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 3/36 2024-05-15T12:19:18.782 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 4/36 2024-05-15T12:19:18.782 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 5/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9. 6/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-15T12:19:18.783 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-15T12:19:18.784 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-15T12:19:18.784 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-15T12:19:18.784 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-15T12:19:18.784 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-15T12:19:18.784 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-15T12:19:18.784 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-15T12:19:18.784 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-15T12:19:18.785 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-15T12:19:18.785 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-15T12:19:18.786 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-15T12:19:18.786 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-15T12:19:18.786 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-15T12:19:18.786 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-15T12:19:18.786 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-15T12:19:18.786 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-15T12:19:18.786 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-15T12:19:18.786 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-15T12:19:18.849 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-15T12:19:19.052 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-15T12:19:19.105 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-15T12:19:19.186 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-15T12:19:19.274 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-15T12:19:19.320 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-15T12:19:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-15T12:19:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9. 31/36 2024-05-15T12:19:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 32/36 2024-05-15T12:19:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 32/36 2024-05-15T12:19:19.666 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:19:19.666 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-15T12:19:19.666 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-15T12:19:19.667 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-15T12:19:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:19.676 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-15T12:19:19.676 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: lua-5.4.4-4.el9.x86_64 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-15T12:19:19.677 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply-3.11-14.el9.noarch 2024-05-15T12:19:19.678 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:19.679 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:19:20.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base 2024-05-15T12:19:20.620 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:06 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:19:20.700 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:19:20.742 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:19:20.743 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T12:19:20.743 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:19:20.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install cephadm 2024-05-15T12:19:21.181 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 33/36 2024-05-15T12:19:21.211 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 33/36 2024-05-15T12:19:21.212 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:19:21.212 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-15T12:19:21.212 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-15T12:19:21.212 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-15T12:19:21.212 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:21.377 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:07 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:19:21.499 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:19:21.499 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:21.499 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 749 k 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 749 k 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 755 k 2024-05-15T12:19:21.500 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:19:21.673 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 34/36 2024-05-15T12:19:21.699 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 34/36 2024-05-15T12:19:21.699 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:19:21.699 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-15T12:19:21.699 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-15T12:19:21.699 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-15T12:19:21.699 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:21.924 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 35/36 2024-05-15T12:19:21.932 INFO:teuthology.orchestra.run.smithi088.stdout:cephadm-19.0.0-1882.gae248125.el9.noarch.rpm 1.7 MB/s | 749 kB 00:00 2024-05-15T12:19:21.933 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:21.933 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.7 MB/s | 749 kB 00:00 2024-05-15T12:19:21.933 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:19:21.938 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:19:21.938 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:19:21.944 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:19:21.944 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:19:21.949 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 35/36 2024-05-15T12:19:21.949 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:19:21.949 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-15T12:19:21.949 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-15T12:19:21.950 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-15T12:19:21.950 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:22.007 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:19.0.0-1882.gae248125.el9.x86_64 36/36 2024-05-15T12:19:22.018 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:19:22.475 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:19:22.561 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:19:22.932 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:19:23.181 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:19:23.182 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:23.182 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:19:23.182 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:19:23.182 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:23.182 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:19:23.386 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-immutable-object-cache 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-2:19.0.0-1882.gae248125.el9.x86_64 36/36 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:19.0.0-1882.gae248125.el9.x86_64 1/36 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 2/36 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 3/36 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 4/36 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 5/36 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9. 6/36 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-15T12:19:23.914 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-15T12:19:23.915 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-15T12:19:23.917 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-15T12:19:23.917 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-15T12:19:23.917 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-15T12:19:23.917 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-15T12:19:23.917 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-15T12:19:23.917 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-15T12:19:23.917 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-15T12:19:23.917 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-15T12:19:23.917 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-15T12:19:23.917 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-15T12:19:23.964 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:09 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:19:24.088 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:19:24.089 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:24.089 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:19:24.089 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:24.089 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:19:24.089 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1882.gae248125.el9 ceph 145 k 2024-05-15T12:19:24.089 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:24.089 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:19:24.089 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:24.089 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:19:24.089 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:24.090 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 145 k 2024-05-15T12:19:24.090 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 436 k 2024-05-15T12:19:24.090 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:19:24.298 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-immutable-object-cache-19.0.0-1882.gae2481 698 kB/s | 145 kB 00:00 2024-05-15T12:19:24.298 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:24.299 INFO:teuthology.orchestra.run.smithi088.stdout:Total 693 kB/s | 145 kB 00:00 2024-05-15T12:19:24.299 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:19:24.306 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:19:24.306 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:19:24.363 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:19:24.363 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:19:24.512 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:19:24.599 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:19:24.624 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:19:24.624 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:19:24.624 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-15T12:19:24.624 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-15T12:19:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:24.940 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:19:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:19:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:24.940 INFO:teuthology.orchestra.run.smithi049.stdout: lua-5.4.4-4.el9.x86_64 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-15T12:19:24.941 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply-3.11-14.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-15T12:19:24.942 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-15T12:19:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:24.943 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:19:25.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base 2024-05-15T12:19:25.531 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:19:25.531 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:25.531 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:19:25.531 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:25.531 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:25.531 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:19:25.773 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr 2024-05-15T12:19:25.894 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:13 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:19:25.978 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:19:26.022 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:19:26.024 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-05-15T12:19:26.024 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:19:26.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install cephadm 2024-05-15T12:19:26.360 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:12 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:19:26.440 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:19:26.483 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:19:26.483 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T12:19:26.483 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:19:26.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-dashboard 2024-05-15T12:19:26.708 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:14 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:19:26.840 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:19:26.841 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:26.841 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-05-15T12:19:26.841 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:26.841 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:19:26.841 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 749 k 2024-05-15T12:19:26.841 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:26.842 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:19:26.842 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:26.842 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:19:26.842 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:26.842 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 749 k 2024-05-15T12:19:26.842 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 755 k 2024-05-15T12:19:26.842 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:19:27.120 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:13 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:19:27.134 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm-19.0.0-1882.gae248125.el9.noarch.rpm 2.5 MB/s | 749 kB 00:00 2024-05-15T12:19:27.135 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:27.136 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.5 MB/s | 749 kB 00:00 2024-05-15T12:19:27.136 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:19:27.146 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:19:27.146 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:19:27.155 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:19:27.156 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:19:27.237 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:19:27.243 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:19:27.244 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:27.244 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 1.8 M 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 31 k 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 16 k 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:27.245 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:19:27.246 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:19:27.246 INFO:teuthology.orchestra.run.smithi088.stdout:Install 12 Packages 2024-05-15T12:19:27.246 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:27.246 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 6.9 M 2024-05-15T12:19:27.246 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 26 M 2024-05-15T12:19:27.246 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:19:27.694 INFO:teuthology.orchestra.run.smithi088.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1882.gae2 83 kB/s | 16 kB 00:00 2024-05-15T12:19:27.727 INFO:teuthology.orchestra.run.smithi088.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1882.gae 136 kB/s | 31 kB 00:00 2024-05-15T12:19:27.743 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:19:27.849 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:19:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:19:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/1 2024-05-15T12:19:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:28.461 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:19:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:19:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:28.461 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:19:28.569 INFO:teuthology.orchestra.run.smithi088.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 320 kB/s | 269 kB 00:00 2024-05-15T12:19:28.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache 2024-05-15T12:19:28.686 INFO:teuthology.orchestra.run.smithi088.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:00 2024-05-15T12:19:28.753 INFO:teuthology.orchestra.run.smithi088.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.6 MB/s | 863 kB 00:00 2024-05-15T12:19:29.028 INFO:teuthology.orchestra.run.smithi088.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-05-15T12:19:29.266 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:17 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:19:29.387 INFO:teuthology.orchestra.run.smithi088.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 783 kB/s | 548 kB 00:00 2024-05-15T12:19:29.394 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:19:29.395 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:29.395 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:19:29.395 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:29.395 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:19:29.395 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1882.gae248125.el9 ceph 145 k 2024-05-15T12:19:29.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:29.395 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:19:29.395 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:29.395 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:19:29.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:29.396 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 145 k 2024-05-15T12:19:29.396 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 436 k 2024-05-15T12:19:29.396 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:19:29.521 INFO:teuthology.orchestra.run.smithi088.stdout:(8/12): ceph-mgr-dashboard-19.0.0-1882.gae24812 902 kB/s | 1.8 MB 00:02 2024-05-15T12:19:29.546 INFO:teuthology.orchestra.run.smithi088.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 911 kB/s | 144 kB 00:00 2024-05-15T12:19:29.671 INFO:teuthology.orchestra.run.smithi088.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-05-15T12:19:29.730 INFO:teuthology.orchestra.run.smithi088.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.9 MB/s | 2.0 MB 00:00 2024-05-15T12:19:29.830 INFO:teuthology.orchestra.run.smithi088.stdout:(12/12): python3-repoze-lru-0.7-16.el9.noarch.r 100 kB/s | 31 kB 00:00 2024-05-15T12:19:29.832 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:29.832 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.7 MB/s | 6.9 MB 00:02 2024-05-15T12:19:29.857 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-19.0.0-1882.gae2481 315 kB/s | 145 kB 00:00 2024-05-15T12:19:29.857 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:29.858 INFO:teuthology.orchestra.run.smithi049.stdout:Total 314 kB/s | 145 kB 00:00 2024-05-15T12:19:29.858 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:19:29.866 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:19:29.866 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:19:29.887 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:19:29.905 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:19:29.905 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:19:29.927 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:19:29.927 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:19:30.077 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:19:30.078 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:19:30.089 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:19:30.149 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:19:30.176 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:19:30.177 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:19:30.177 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-15T12:19:30.177 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:30.688 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:19:30.765 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-15T12:19:30.799 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-15T12:19:30.847 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-15T12:19:30.885 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-15T12:19:30.990 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-15T12:19:31.120 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-15T12:19:31.147 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1882.gae248125. 1/1 2024-05-15T12:19:31.147 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:31.147 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:19:31.147 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:19:31.147 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:31.147 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:19:31.210 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-15T12:19:31.266 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-15T12:19:31.293 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-15T12:19:31.333 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9 10/12 2024-05-15T12:19:31.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr 2024-05-15T12:19:31.580 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el 11/12 2024-05-15T12:19:31.608 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 12/12 2024-05-15T12:19:31.931 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:19 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:19:32.018 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-mgr-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:19:32.063 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:19:32.065 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-05-15T12:19:32.065 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:19:32.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-dashboard 2024-05-15T12:19:32.395 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 12/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el 1/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 2/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9 3/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-15T12:19:32.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-15T12:19:32.704 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-15T12:19:32.704 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:32.704 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:19:32.704 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-15T12:19:32.704 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:32.705 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:19:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:20 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:19:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:19:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:32.869 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-05-15T12:19:32.869 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 1.8 M 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 31 k 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 16 k 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-15T12:19:32.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-15T12:19:32.871 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:19:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:19:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Install 12 Packages 2024-05-15T12:19:32.871 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 6.9 M 2024-05-15T12:19:32.871 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 26 M 2024-05-15T12:19:32.872 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:19:32.941 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-15T12:19:33.460 INFO:teuthology.orchestra.run.smithi049.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1882.gae 103 kB/s | 31 kB 00:00 2024-05-15T12:19:33.486 INFO:teuthology.orchestra.run.smithi049.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1882.gae2 50 kB/s | 16 kB 00:00 2024-05-15T12:19:33.510 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:19 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:19:33.569 INFO:teuthology.orchestra.run.smithi049.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.2 MB/s | 269 kB 00:00 2024-05-15T12:19:33.619 INFO:teuthology.orchestra.run.smithi049.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-05-15T12:19:33.630 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:19:33.631 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================= 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================= 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 7.4 M 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-15T12:19:33.632 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-15T12:19:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-15T12:19:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-15T12:19:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-15T12:19:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-15T12:19:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-15T12:19:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:33.633 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:19:33.633 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================= 2024-05-15T12:19:33.633 INFO:teuthology.orchestra.run.smithi088.stdout:Install 12 Packages 2024-05-15T12:19:33.633 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:19:33.634 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 38 M 2024-05-15T12:19:33.634 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 209 M 2024-05-15T12:19:33.634 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:19:33.803 INFO:teuthology.orchestra.run.smithi049.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.9 MB/s | 548 kB 00:00 2024-05-15T12:19:33.845 INFO:teuthology.orchestra.run.smithi049.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 465 kB/s | 19 kB 00:00 2024-05-15T12:19:33.953 INFO:teuthology.orchestra.run.smithi049.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-05-15T12:19:33.987 INFO:teuthology.orchestra.run.smithi049.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.0 MB/s | 863 kB 00:00 2024-05-15T12:19:34.012 INFO:teuthology.orchestra.run.smithi049.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-05-15T12:19:34.054 INFO:teuthology.orchestra.run.smithi049.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-05-15T12:19:34.071 INFO:teuthology.orchestra.run.smithi049.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 366 kB/s | 31 kB 00:00 2024-05-15T12:19:34.222 INFO:teuthology.orchestra.run.smithi049.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1882.gae2481 1.7 MB/s | 1.8 MB 00:01 2024-05-15T12:19:34.224 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:34.224 INFO:teuthology.orchestra.run.smithi049.stdout:Total 5.1 MB/s | 6.9 MB 00:01 2024-05-15T12:19:34.275 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:19:34.293 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:19:34.293 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:19:34.370 INFO:teuthology.orchestra.run.smithi088.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 367 kB/s | 195 kB 00:00 2024-05-15T12:19:34.477 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:19:34.477 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:19:34.487 INFO:teuthology.orchestra.run.smithi088.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-05-15T12:19:34.854 INFO:teuthology.orchestra.run.smithi088.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 790 kB/s | 803 kB 00:01 2024-05-15T12:19:34.954 INFO:teuthology.orchestra.run.smithi088.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 182 kB/s | 18 kB 00:00 2024-05-15T12:19:35.055 INFO:teuthology.orchestra.run.smithi088.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 352 kB/s | 35 kB 00:00 2024-05-15T12:19:35.091 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:19:35.182 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-15T12:19:35.205 INFO:teuthology.orchestra.run.smithi088.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.2 MB/s | 3.0 MB 00:00 2024-05-15T12:19:35.225 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-15T12:19:35.273 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-15T12:19:35.297 INFO:teuthology.orchestra.run.smithi088.stdout:(7/12): ceph-mgr-diskprediction-local-19.0.0-18 5.1 MB/s | 7.4 MB 00:01 2024-05-15T12:19:35.310 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-15T12:19:35.435 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-15T12:19:35.564 INFO:teuthology.orchestra.run.smithi088.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 681 kB/s | 244 kB 00:00 2024-05-15T12:19:35.575 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-15T12:19:35.666 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-15T12:19:35.716 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-15T12:19:35.743 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-15T12:19:35.775 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9 10/12 2024-05-15T12:19:35.931 INFO:teuthology.orchestra.run.smithi088.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 683 kB/s | 250 kB 00:00 2024-05-15T12:19:36.031 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el 11/12 2024-05-15T12:19:36.058 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 12/12 2024-05-15T12:19:36.825 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 12/12 2024-05-15T12:19:36.825 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el 1/12 2024-05-15T12:19:36.825 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noa 2/12 2024-05-15T12:19:36.826 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9 3/12 2024-05-15T12:19:36.826 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-15T12:19:36.826 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-15T12:19:36.826 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-15T12:19:36.826 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-15T12:19:36.826 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-15T12:19:36.826 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-15T12:19:36.826 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-15T12:19:36.826 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-15T12:19:37.072 INFO:teuthology.orchestra.run.smithi088.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.3 MB/s | 4.7 MB 00:02 2024-05-15T12:19:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-15T12:19:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:19:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-15T12:19:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:19:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:19:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:19:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-15T12:19:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-15T12:19:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-15T12:19:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-15T12:19:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-15T12:19:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-15T12:19:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-15T12:19:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-15T12:19:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:37.126 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:19:37.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-15T12:19:37.481 INFO:teuthology.orchestra.run.smithi088.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.3 MB/s | 5.1 MB 00:02 2024-05-15T12:19:37.946 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:25 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:19:38.077 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:19:38.078 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-05-15T12:19:38.078 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-05-15T12:19:38.078 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-05-15T12:19:38.078 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 7.4 M 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-15T12:19:38.079 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-15T12:19:38.080 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:38.080 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:19:38.080 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-05-15T12:19:38.080 INFO:teuthology.orchestra.run.smithi049.stdout:Install 12 Packages 2024-05-15T12:19:38.080 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:19:38.080 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 38 M 2024-05-15T12:19:38.080 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 209 M 2024-05-15T12:19:38.080 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:19:39.070 INFO:teuthology.orchestra.run.smithi049.stdout:(1/12): ceph-mgr-diskprediction-local-19.0.0-18 13 MB/s | 7.4 MB 00:00 2024-05-15T12:19:39.336 INFO:teuthology.orchestra.run.smithi049.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 123 kB/s | 33 kB 00:00 2024-05-15T12:19:39.370 INFO:teuthology.orchestra.run.smithi049.stdout:(3/12): libquadmath-11.4.1-3.el9.x86_64.rpm 222 kB/s | 195 kB 00:00 2024-05-15T12:19:39.512 INFO:teuthology.orchestra.run.smithi049.stdout:(4/12): libgfortran-11.4.1-3.el9.x86_64.rpm 787 kB/s | 803 kB 00:01 2024-05-15T12:19:39.629 INFO:teuthology.orchestra.run.smithi049.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 70 kB/s | 18 kB 00:00 2024-05-15T12:19:39.779 INFO:teuthology.orchestra.run.smithi049.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 132 kB/s | 35 kB 00:00 2024-05-15T12:19:40.230 INFO:teuthology.orchestra.run.smithi049.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.3 MB/s | 3.0 MB 00:00 2024-05-15T12:19:40.255 INFO:teuthology.orchestra.run.smithi049.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 513 kB/s | 244 kB 00:00 2024-05-15T12:19:40.447 INFO:teuthology.orchestra.run.smithi049.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 250 kB 00:00 2024-05-15T12:19:40.747 INFO:teuthology.orchestra.run.smithi049.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.9 MB/s | 5.1 MB 00:00 2024-05-15T12:19:40.814 INFO:teuthology.orchestra.run.smithi049.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 4.0 MB/s | 4.7 MB 00:01 2024-05-15T12:19:54.586 INFO:teuthology.orchestra.run.smithi088.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 892 kB/s | 16 MB 00:18 2024-05-15T12:19:54.587 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:54.588 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.8 MB/s | 38 MB 00:20 2024-05-15T12:19:54.837 INFO:teuthology.orchestra.run.smithi049.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.1 MB/s | 16 MB 00:14 2024-05-15T12:19:54.838 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:19:54.838 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.3 MB/s | 38 MB 00:16 2024-05-15T12:19:54.942 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:19:54.968 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:19:54.968 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:19:55.209 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:19:55.236 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:19:55.236 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:19:55.396 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:19:55.397 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:19:55.687 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:19:55.688 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:19:56.147 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:19:56.203 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-15T12:19:56.243 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-15T12:19:56.341 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-15T12:19:56.385 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-15T12:19:56.474 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:19:56.546 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-15T12:19:56.563 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-15T12:19:56.586 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-15T12:19:56.593 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-15T12:19:56.669 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-15T12:19:56.689 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-15T12:19:56.736 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-15T12:19:56.921 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-15T12:19:56.951 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-15T12:19:57.036 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-15T12:19:57.325 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-15T12:19:57.456 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-15T12:19:57.718 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-15T12:19:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-15T12:19:58.674 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-15T12:19:59.156 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-15T12:19:59.697 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-15T12:19:59.734 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 12/12 2024-05-15T12:20:00.241 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-15T12:20:00.267 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 12/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 12/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 1/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-15T12:20:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-15T12:20:02.634 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-15T12:20:02.634 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:02.634 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:02.634 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:20:02.634 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-15T12:20:02.634 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-15T12:20:02.634 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-15T12:20:02.634 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-15T12:20:02.634 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-15T12:20:02.634 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-15T12:20:02.635 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-15T12:20:02.635 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-15T12:20:02.635 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-15T12:20:02.635 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-15T12:20:02.635 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-15T12:20:02.635 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:02.635 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:02.724 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 12/12 2024-05-15T12:20:02.724 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248 1/12 2024-05-15T12:20:02.724 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-15T12:20:02.724 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-15T12:20:02.724 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-15T12:20:02.724 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-15T12:20:02.724 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-15T12:20:02.724 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-15T12:20:02.725 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-15T12:20:02.725 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-15T12:20:02.725 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-15T12:20:02.725 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-15T12:20:03.008 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-rook 2024-05-15T12:20:03.221 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-15T12:20:03.222 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-15T12:20:03.223 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-15T12:20:03.223 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-15T12:20:03.223 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-15T12:20:03.223 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:03.223 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:03.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-rook 2024-05-15T12:20:03.590 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:49 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:03.712 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:03.713 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T12:20:03.713 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:03.713 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:03.713 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 49 k 2024-05-15T12:20:03.713 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:03.714 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:03.715 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:03.715 INFO:teuthology.orchestra.run.smithi088.stdout:Install 13 Packages 2024-05-15T12:20:03.715 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:03.715 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 2.2 M 2024-05-15T12:20:03.715 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 26 M 2024-05-15T12:20:03.715 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:04.133 INFO:teuthology.orchestra.run.smithi088.stdout:(1/13): ceph-mgr-rook-19.0.0-1882.gae248125.el9 162 kB/s | 49 kB 00:00 2024-05-15T12:20:04.179 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:52 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:04.308 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 49 k 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-15T12:20:04.310 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout:Install 13 Packages 2024-05-15T12:20:04.311 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:04.312 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.2 M 2024-05-15T12:20:04.312 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 26 M 2024-05-15T12:20:04.312 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:20:04.367 INFO:teuthology.orchestra.run.smithi088.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 36 kB/s | 19 kB 00:00 2024-05-15T12:20:04.467 INFO:teuthology.orchestra.run.smithi088.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 40 kB/s | 26 kB 00:00 2024-05-15T12:20:04.709 INFO:teuthology.orchestra.run.smithi049.stdout:(1/13): ceph-mgr-rook-19.0.0-1882.gae248125.el9 156 kB/s | 49 kB 00:00 2024-05-15T12:20:04.942 INFO:teuthology.orchestra.run.smithi088.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 275 kB/s | 222 kB 00:00 2024-05-15T12:20:04.943 INFO:teuthology.orchestra.run.smithi049.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 35 kB/s | 19 kB 00:00 2024-05-15T12:20:05.018 INFO:teuthology.orchestra.run.smithi049.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 41 kB/s | 26 kB 00:00 2024-05-15T12:20:05.184 INFO:teuthology.orchestra.run.smithi088.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 200 kB/s | 159 kB 00:00 2024-05-15T12:20:05.301 INFO:teuthology.orchestra.run.smithi088.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 348 kB/s | 32 kB 00:00 2024-05-15T12:20:05.326 INFO:teuthology.orchestra.run.smithi088.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 325 kB/s | 279 kB 00:00 2024-05-15T12:20:05.351 INFO:teuthology.orchestra.run.smithi088.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 305 kB/s | 15 kB 00:00 2024-05-15T12:20:05.502 INFO:teuthology.orchestra.run.smithi088.stdout:(9/13): python3-google-auth-2.29.0-1.el9.noarch 1.1 MB/s | 206 kB 00:00 2024-05-15T12:20:05.535 INFO:teuthology.orchestra.run.smithi049.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 269 kB/s | 222 kB 00:00 2024-05-15T12:20:05.552 INFO:teuthology.orchestra.run.smithi088.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-15T12:20:05.614 INFO:teuthology.orchestra.run.smithi088.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-05-15T12:20:05.636 INFO:teuthology.orchestra.run.smithi088.stdout:(12/13): python3-requests-oauthlib-1.3.0-12.el9 77 kB/s | 54 kB 00:00 2024-05-15T12:20:05.669 INFO:teuthology.orchestra.run.smithi088.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.2 MB/s | 1.0 MB 00:00 2024-05-15T12:20:05.671 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:05.671 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.1 MB/s | 2.2 MB 00:01 2024-05-15T12:20:05.727 INFO:teuthology.orchestra.run.smithi049.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 209 kB/s | 159 kB 00:00 2024-05-15T12:20:05.734 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:20:05.749 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:20:05.750 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:20:05.834 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:20:05.836 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:20:05.844 INFO:teuthology.orchestra.run.smithi049.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 338 kB/s | 279 kB 00:00 2024-05-15T12:20:05.944 INFO:teuthology.orchestra.run.smithi049.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 149 kB/s | 32 kB 00:00 2024-05-15T12:20:06.036 INFO:teuthology.orchestra.run.smithi049.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-05-15T12:20:06.070 INFO:teuthology.orchestra.run.smithi049.stdout:(9/13): python3-google-auth-2.29.0-1.el9.noarch 1.6 MB/s | 206 kB 00:00 2024-05-15T12:20:06.128 INFO:teuthology.orchestra.run.smithi049.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-05-15T12:20:06.187 INFO:teuthology.orchestra.run.smithi049.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-05-15T12:20:06.212 INFO:teuthology.orchestra.run.smithi049.stdout:(12/13): python3-requests-oauthlib-1.3.0-12.el9 79 kB/s | 54 kB 00:00 2024-05-15T12:20:06.287 INFO:teuthology.orchestra.run.smithi049.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-05-15T12:20:06.290 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:06.290 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.1 MB/s | 2.2 MB 00:01 2024-05-15T12:20:06.334 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:20:06.359 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:20:06.375 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:20:06.375 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:20:06.471 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:20:06.472 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:20:06.500 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-15T12:20:06.554 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-15T12:20:06.609 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-15T12:20:06.646 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-15T12:20:06.688 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-15T12:20:06.781 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-15T12:20:06.885 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-15T12:20:06.948 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-15T12:20:06.988 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:20:07.185 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-15T12:20:07.247 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-15T12:20:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-15T12:20:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-15T12:20:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-15T12:20:07.486 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-15T12:20:07.597 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-15T12:20:07.657 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-15T12:20:07.673 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-15T12:20:07.728 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-15T12:20:07.762 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-15T12:20:07.809 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-15T12:20:07.831 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 13/13 2024-05-15T12:20:08.423 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-15T12:20:08.469 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-15T12:20:08.503 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-15T12:20:08.546 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 13/13 2024-05-15T12:20:08.546 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 1/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-15T12:20:08.547 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-15T12:20:08.552 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-15T12:20:08.580 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 13/13 2024-05-15T12:20:08.871 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-15T12:20:08.871 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:08.871 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:08.871 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:20:08.871 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-15T12:20:08.871 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:08.872 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:09.148 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-cephadm 2024-05-15T12:20:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 13/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 1/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-15T12:20:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-15T12:20:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-15T12:20:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:09.672 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:20:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:09.673 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:09.723 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:55 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:09.845 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:09.845 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 138 k 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:09.846 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:09.847 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2024-05-15T12:20:09.847 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:09.847 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 7.0 M 2024-05-15T12:20:09.847 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 33 M 2024-05-15T12:20:09.847 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:09.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-cephadm 2024-05-15T12:20:10.512 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:58 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:10.537 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1882.gae248125.e 308 kB/s | 138 kB 00:00 2024-05-15T12:20:10.641 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 138 k 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-15T12:20:10.642 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:10.643 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:20:10.643 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:10.643 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2024-05-15T12:20:10.643 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:10.643 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 7.0 M 2024-05-15T12:20:10.643 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 33 M 2024-05-15T12:20:10.643 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:20:10.796 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-05-15T12:20:10.863 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 868 kB/s | 58 kB 00:00 2024-05-15T12:20:10.947 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 612 kB/s | 51 kB 00:00 2024-05-15T12:20:11.097 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 246 kB/s | 248 kB 00:01 2024-05-15T12:20:11.378 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 694 kB/s | 248 kB 00:00 2024-05-15T12:20:11.470 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1882.gae248125.e 308 kB/s | 138 kB 00:00 2024-05-15T12:20:11.614 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-05-15T12:20:11.614 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:11.615 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.9 MB/s | 7.0 MB 00:01 2024-05-15T12:20:11.685 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:20:11.697 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:20:11.697 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:20:11.794 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:20:11.794 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:20:11.845 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 154 kB/s | 58 kB 00:00 2024-05-15T12:20:11.987 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 360 kB/s | 51 kB 00:00 2024-05-15T12:20:12.045 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 821 kB/s | 548 kB 00:00 2024-05-15T12:20:12.102 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:20:12.271 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-15T12:20:12.323 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-15T12:20:12.763 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.4 MB/s | 6.0 MB 00:01 2024-05-15T12:20:12.764 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:12.764 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.3 MB/s | 7.0 MB 00:02 2024-05-15T12:20:12.838 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-15T12:20:12.840 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:20:12.853 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:20:12.853 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:20:12.918 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-15T12:20:12.957 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:20:12.957 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:20:12.988 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-15T12:20:13.019 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 6/6 2024-05-15T12:20:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:20:13.446 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-15T12:20:13.499 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-15T12:20:13.755 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 6/6 2024-05-15T12:20:13.756 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/6 2024-05-15T12:20:13.756 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-15T12:20:13.756 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-15T12:20:13.756 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-15T12:20:13.756 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:14.012 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:14.061 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-15T12:20:14.146 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-15T12:20:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-15T12:20:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 6/6 2024-05-15T12:20:14.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-fuse 2024-05-15T12:20:14.831 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:00 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:14.954 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:14.954 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:14.954 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T12:20:14.954 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:14.954 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:14.955 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:19.0.0-1882.gae248125.el9 ceph 888 k 2024-05-15T12:20:14.955 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:20:14.955 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-15T12:20:14.955 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:14.955 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:14.955 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:14.955 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-05-15T12:20:14.955 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:14.956 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 968 k 2024-05-15T12:20:14.956 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.8 M 2024-05-15T12:20:14.956 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:15.062 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 6/6 2024-05-15T12:20:15.063 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 1/6 2024-05-15T12:20:15.063 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-15T12:20:15.063 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-15T12:20:15.063 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-15T12:20:15.063 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-15T12:20:15.132 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 724 kB/s | 80 kB 00:00 2024-05-15T12:20:15.332 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-15T12:20:15.332 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:15.332 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:20:15.332 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:20:15.332 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-15T12:20:15.332 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-15T12:20:15.332 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-15T12:20:15.332 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-15T12:20:15.332 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-15T12:20:15.333 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:15.333 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:15.465 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-fuse-19.0.0-1882.gae248125.el9.x86_ 2.0 MB/s | 888 kB 00:00 2024-05-15T12:20:15.467 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:15.467 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.9 MB/s | 968 kB 00:00 2024-05-15T12:20:15.478 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:20:15.501 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:20:15.501 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:20:15.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse 2024-05-15T12:20:15.579 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:20:15.579 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:20:15.773 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:20:15.900 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-15T12:20:15.974 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:20:16.175 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:04 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:16.300 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:16.300 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:16.300 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-05-15T12:20:16.300 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:16.300 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:20:16.300 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:19.0.0-1882.gae248125.el9 ceph 888 k 2024-05-15T12:20:16.301 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:20:16.301 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-15T12:20:16.301 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:16.301 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:20:16.301 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:16.301 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2024-05-15T12:20:16.301 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:16.301 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 968 k 2024-05-15T12:20:16.301 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 2.8 M 2024-05-15T12:20:16.302 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:20:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:20:16.629 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/2 2024-05-15T12:20:16.730 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 223 kB/s | 80 kB 00:00 2024-05-15T12:20:16.864 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): ceph-fuse-19.0.0-1882.gae248125.el9.x86_ 1.8 MB/s | 888 kB 00:00 2024-05-15T12:20:16.865 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:16.865 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.7 MB/s | 968 kB 00:00 2024-05-15T12:20:16.878 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:20:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-15T12:20:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:16.886 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-15T12:20:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:16.886 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:16.901 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:20:16.902 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:20:16.984 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:20:16.984 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:20:17.177 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-volume 2024-05-15T12:20:17.305 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:20:17.453 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-15T12:20:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:20:17.762 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:03 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:17.888 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 264 k 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-05-15T12:20:17.889 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:17.890 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 341 k 2024-05-15T12:20:17.890 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.5 M 2024-05-15T12:20:17.890 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:18.220 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 425 kB/s | 77 kB 00:00 2024-05-15T12:20:18.329 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:20:18.329 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/2 2024-05-15T12:20:18.420 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-volume-19.0.0-1882.gae248125.el9.no 691 kB/s | 264 kB 00:00 2024-05-15T12:20:18.420 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:18.421 INFO:teuthology.orchestra.run.smithi088.stdout:Total 642 kB/s | 341 kB 00:00 2024-05-15T12:20:18.433 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:20:18.442 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:20:18.442 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:20:18.474 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:20:18.474 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:20:18.597 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-15T12:20:18.597 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:18.597 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:20:18.597 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:19.0.0-1882.gae248125.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-15T12:20:18.597 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:18.597 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:18.615 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:20:18.794 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-15T12:20:18.839 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2/2 2024-05-15T12:20:18.855 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2/2 2024-05-15T12:20:18.855 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:20:18.855 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-15T12:20:18.855 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:18.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-volume 2024-05-15T12:20:19.467 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:07 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:19.594 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:19.595 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-05-15T12:20:19.595 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:19.595 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:20:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume noarch 2:19.0.0-1882.gae248125.el9 ceph-noarch 264 k 2024-05-15T12:20:19.595 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:20:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-15T12:20:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:19.596 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:20:19.596 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:19.596 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2024-05-15T12:20:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:19.596 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 341 k 2024-05-15T12:20:19.596 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.5 M 2024-05-15T12:20:19.596 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:20:19.597 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 1/2 2024-05-15T12:20:19.886 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-15T12:20:19.886 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:19.886 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:19.886 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:20:19.886 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-15T12:20:19.886 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:19.886 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:20.044 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): ceph-volume-19.0.0-1882.gae248125.el9.no 663 kB/s | 264 kB 00:00 2024-05-15T12:20:20.184 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados-devel 2024-05-15T12:20:20.327 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 113 kB/s | 77 kB 00:00 2024-05-15T12:20:20.328 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:20.328 INFO:teuthology.orchestra.run.smithi049.stdout:Total 465 kB/s | 341 kB 00:00 2024-05-15T12:20:20.341 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:20:20.352 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:20:20.353 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:20:20.386 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:20:20.386 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:20:20.539 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:20:20.736 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-15T12:20:20.771 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2/2 2024-05-15T12:20:20.776 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:06 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:20.788 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2/2 2024-05-15T12:20:20.789 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:20:20.789 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-15T12:20:20.789 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:20.897 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:20.898 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:20.898 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:20:20.898 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:20.898 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:20.898 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:19.0.0-1882.gae248125.el9 ceph 127 k 2024-05-15T12:20:20.898 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:20.898 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:20.898 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:20.898 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:20:20.898 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:20.899 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 127 k 2024-05-15T12:20:20.899 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 456 k 2024-05-15T12:20:20.899 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:21.117 INFO:teuthology.orchestra.run.smithi088.stdout:librados-devel-19.0.0-1882.gae248125.el9.x86_64 582 kB/s | 127 kB 00:00 2024-05-15T12:20:21.117 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:21.118 INFO:teuthology.orchestra.run.smithi088.stdout:Total 578 kB/s | 127 kB 00:00 2024-05-15T12:20:21.118 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:20:21.123 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:20:21.124 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:20:21.187 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:20:21.187 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:20:21.339 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:20:21.447 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:21.517 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 1/2 2024-05-15T12:20:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-15T12:20:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:21.813 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:20:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume-2:19.0.0-1882.gae248125.el9.noarch 2024-05-15T12:20:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-15T12:20:21.814 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:21.814 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:21.876 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:22.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel 2024-05-15T12:20:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:22.127 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:22.128 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:22.128 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:22.128 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:22.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs2 2024-05-15T12:20:22.736 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:10 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:22.864 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:22.865 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:20:22.865 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:22.865 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:20:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:19.0.0-1882.gae248125.el9 ceph 127 k 2024-05-15T12:20:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:22.866 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:20:22.866 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:22.866 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:20:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:22.866 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 127 k 2024-05-15T12:20:22.866 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 456 k 2024-05-15T12:20:22.866 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:20:22.963 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:08 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:23.046 INFO:teuthology.orchestra.run.smithi088.stdout:Package libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:23.088 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:23.089 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T12:20:23.089 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:23.166 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs-devel 2024-05-15T12:20:23.241 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-19.0.0-1882.gae248125.el9.x86_64 338 kB/s | 127 kB 00:00 2024-05-15T12:20:23.242 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:23.242 INFO:teuthology.orchestra.run.smithi049.stdout:Total 337 kB/s | 127 kB 00:00 2024-05-15T12:20:23.242 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:20:23.248 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:20:23.248 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:20:23.316 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:20:23.316 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:20:23.471 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:20:23.576 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:23.747 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:09 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:23.871 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:23.872 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:23.872 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:20:23.872 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:23.872 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:23.872 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:19.0.0-1882.gae248125.el9 ceph 31 k 2024-05-15T12:20:23.872 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:23.872 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:23.872 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:23.872 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:20:23.872 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:23.873 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 31 k 2024-05-15T12:20:23.873 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 140 k 2024-05-15T12:20:23.873 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:23.981 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:24.037 INFO:teuthology.orchestra.run.smithi088.stdout:libcephfs-devel-19.0.0-1882.gae248125.el9.x86_6 190 kB/s | 31 kB 00:00 2024-05-15T12:20:24.037 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:24.038 INFO:teuthology.orchestra.run.smithi088.stdout:Total 189 kB/s | 31 kB 00:00 2024-05-15T12:20:24.038 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:20:24.041 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:20:24.041 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:20:24.065 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:20:24.065 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:20:24.143 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:20:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:24.229 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:20:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:24.229 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:24.242 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:24.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2 2024-05-15T12:20:24.619 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:24.834 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:24.835 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:24.835 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:24.835 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:25.018 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados2 2024-05-15T12:20:25.098 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:13 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:25.182 INFO:teuthology.orchestra.run.smithi049.stdout:Package libcephfs2-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:25.226 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:25.228 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-05-15T12:20:25.228 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:25.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs-devel 2024-05-15T12:20:25.597 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:11 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:25.677 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:25.718 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:25.719 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T12:20:25.719 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:25.789 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librbd1 2024-05-15T12:20:25.903 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:13 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:26.032 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:26.032 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:19.0.0-1882.gae248125.el9 ceph 31 k 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 31 k 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 140 k 2024-05-15T12:20:26.033 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:20:26.203 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs-devel-19.0.0-1882.gae248125.el9.x86_6 184 kB/s | 31 kB 00:00 2024-05-15T12:20:26.204 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:26.204 INFO:teuthology.orchestra.run.smithi049.stdout:Total 182 kB/s | 31 kB 00:00 2024-05-15T12:20:26.204 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:20:26.207 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:20:26.208 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:20:26.223 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:20:26.224 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:20:26.304 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:20:26.368 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:12 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:26.420 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:26.447 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:26.490 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:26.491 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T12:20:26.491 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:26.564 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rados 2024-05-15T12:20:26.794 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:27.025 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:27.025 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:27.025 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:20:27.025 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:27.025 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:27.026 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:27.141 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:13 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:27.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2 2024-05-15T12:20:27.221 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:27.263 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:27.264 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T12:20:27.264 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:27.338 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rgw 2024-05-15T12:20:27.803 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:27.886 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:27.919 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:13 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:27.930 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:27.932 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-05-15T12:20:27.932 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:27.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1 2024-05-15T12:20:28.001 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:28.043 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:28.044 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T12:20:28.044 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:28.130 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-cephfs 2024-05-15T12:20:28.594 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:16 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:28.677 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:28.713 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:14 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:28.721 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:28.722 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-05-15T12:20:28.722 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:28.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados 2024-05-15T12:20:28.796 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:28.837 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:28.838 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T12:20:28.838 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:28.909 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rbd 2024-05-15T12:20:29.388 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:17 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:29.473 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rados-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:29.483 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:29.517 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:29.518 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-05-15T12:20:29.519 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:29.563 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:29.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw 2024-05-15T12:20:29.604 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:29.605 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T12:20:29.605 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:29.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-fuse 2024-05-15T12:20:30.197 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:18 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:30.258 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:16 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:30.286 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rgw-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:30.331 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:30.332 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-05-15T12:20:30.332 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:30.388 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:30.388 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:30.388 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T12:20:30.389 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:30.389 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:30.389 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:19.0.0-1882.gae248125.el9 ceph 85 k 2024-05-15T12:20:30.389 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:30.389 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:30.389 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:30.389 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:20:30.389 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:30.390 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 85 k 2024-05-15T12:20:30.390 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 231 k 2024-05-15T12:20:30.390 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:30.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs 2024-05-15T12:20:30.607 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-fuse-19.0.0-1882.gae248125.el9.x86_64.rpm 394 kB/s | 85 kB 00:00 2024-05-15T12:20:30.607 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:30.608 INFO:teuthology.orchestra.run.smithi088.stdout:Total 390 kB/s | 85 kB 00:00 2024-05-15T12:20:30.608 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:20:30.615 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:20:30.615 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:20:30.680 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:20:30.680 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:20:30.835 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:20:30.939 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:19 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:31.103 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-cephfs-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:31.147 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:31.148 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-05-15T12:20:31.148 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:31.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd 2024-05-15T12:20:31.314 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:31.586 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:31.586 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:31.586 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:31.586 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:31.586 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:31.586 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:31.818 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:19 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:31.848 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-mirror 2024-05-15T12:20:31.902 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rbd-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:20:31.947 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:31.948 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-05-15T12:20:31.948 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:32.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse 2024-05-15T12:20:32.429 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:18 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:32.556 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:32.557 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:32.557 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:20:32.557 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:32.558 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:32.558 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.1 M 2024-05-15T12:20:32.558 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:32.558 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:32.558 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:32.558 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:20:32.558 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:32.558 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 3.1 M 2024-05-15T12:20:32.558 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 13 M 2024-05-15T12:20:32.558 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:32.612 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:20 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:32.738 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:32.739 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-05-15T12:20:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:20:32.739 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:19.0.0-1882.gae248125.el9 ceph 85 k 2024-05-15T12:20:32.739 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:20:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:32.739 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:20:32.740 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 85 k 2024-05-15T12:20:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 231 k 2024-05-15T12:20:32.740 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:20:32.983 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-mirror-19.0.0-1882.gae248125.el9.x86_64.rpm 7.3 MB/s | 3.1 MB 00:00 2024-05-15T12:20:32.984 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:32.984 INFO:teuthology.orchestra.run.smithi088.stdout:Total 7.2 MB/s | 3.1 MB 00:00 2024-05-15T12:20:32.984 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:20:32.993 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:20:32.993 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:20:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-19.0.0-1882.gae248125.el9.x86_64.rpm 330 kB/s | 85 kB 00:00 2024-05-15T12:20:32.998 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:Total 329 kB/s | 85 kB 00:00 2024-05-15T12:20:32.999 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:20:33.006 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:20:33.007 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:20:33.074 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:20:33.074 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:20:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:20:33.077 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:20:33.232 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:20:33.332 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:33.504 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:20:33.576 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:33.601 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:33.601 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:20:33.601 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-15T12:20:33.601 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-15T12:20:33.601 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-15T12:20:33.602 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:33.714 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:33.966 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:33.966 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:33.966 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:20:33.966 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:33.966 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:33.966 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:34.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror 2024-05-15T12:20:34.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:34.658 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:34.658 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:34.658 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:34.658 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:34.658 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:34.805 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:22 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:34.915 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-nbd 2024-05-15T12:20:34.934 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:34.935 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:34.935 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:20:34.935 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:34.935 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:20:34.935 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:19.0.0-1882.gae248125.el9 ceph 3.1 M 2024-05-15T12:20:34.935 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:34.936 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:20:34.936 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:34.936 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:20:34.936 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:34.936 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 3.1 M 2024-05-15T12:20:34.936 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 13 M 2024-05-15T12:20:34.936 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:20:35.401 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-19.0.0-1882.gae248125.el9.x86_64.rpm 6.6 MB/s | 3.1 MB 00:00 2024-05-15T12:20:35.402 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:35.402 INFO:teuthology.orchestra.run.smithi049.stdout:Total 6.6 MB/s | 3.1 MB 00:00 2024-05-15T12:20:35.402 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:20:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:20:35.412 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:20:35.499 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:20:35.499 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:20:35.501 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:21 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:35.626 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:35.627 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:35.627 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-15T12:20:35.627 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:35.627 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:35.627 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:19.0.0-1882.gae248125.el9 ceph 171 k 2024-05-15T12:20:35.627 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:35.627 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:35.627 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:35.627 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:20:35.627 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:35.628 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 171 k 2024-05-15T12:20:35.628 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 490 k 2024-05-15T12:20:35.628 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:35.946 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:20:36.019 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:36.045 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-15T12:20:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-15T12:20:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-15T12:20:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-15T12:20:36.046 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:36.185 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-nbd-19.0.0-1882.gae248125.el9.x86_64.rpm 307 kB/s | 171 kB 00:00 2024-05-15T12:20:36.185 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:36.185 INFO:teuthology.orchestra.run.smithi088.stdout:Total 306 kB/s | 171 kB 00:00 2024-05-15T12:20:36.185 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:20:36.193 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:20:36.193 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:20:36.257 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:20:36.257 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:20:36.406 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:20:36.500 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:36.949 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:37.147 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:20:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:37.148 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:37.199 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:37.199 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:37.199 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:37.199 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:37.199 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:37.199 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:37.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd 2024-05-15T12:20:37.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base-debuginfo 2024-05-15T12:20:37.991 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:25 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:38.017 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:24 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:38.118 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:19.0.0-1882.gae248125.el9 ceph 171 k 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 171 k 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 490 k 2024-05-15T12:20:38.119 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:20:38.145 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:38.146 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:38.146 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:20:38.146 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:38.146 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:38.146 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 68 M 2024-05-15T12:20:38.146 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-15T12:20:38.146 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 23 M 2024-05-15T12:20:38.146 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:38.146 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:38.146 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:38.147 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-05-15T12:20:38.147 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:38.147 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 91 M 2024-05-15T12:20:38.147 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 501 M 2024-05-15T12:20:38.147 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:38.826 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-19.0.0-1882.gae248125.el9.x86_64.rpm 242 kB/s | 171 kB 00:00 2024-05-15T12:20:38.826 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:38.827 INFO:teuthology.orchestra.run.smithi049.stdout:Total 241 kB/s | 171 kB 00:00 2024-05-15T12:20:38.827 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:20:38.838 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:20:38.838 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:20:38.910 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:20:38.910 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:20:39.073 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:20:39.184 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:39.864 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:20:39.864 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:39.864 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:20:39.864 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:39.865 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:39.865 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:20:40.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base-debuginfo 2024-05-15T12:20:40.699 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:28 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:20:40.830 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:20:40.831 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:40.831 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:20:40.831 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:40.831 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:20:40.831 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 68 M 2024-05-15T12:20:40.831 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-05-15T12:20:40.832 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 23 M 2024-05-15T12:20:40.832 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:40.832 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:20:40.832 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:20:40.832 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2024-05-15T12:20:40.832 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:20:40.832 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 91 M 2024-05-15T12:20:40.832 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 501 M 2024-05-15T12:20:40.833 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:20:42.822 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): ceph-base-debuginfo-19.0.0-1882.gae24812 15 MB/s | 68 MB 00:04 2024-05-15T12:20:43.588 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-debuginfo-19.0.0-1882.gae248125.el9 4.2 MB/s | 23 MB 00:05 2024-05-15T12:20:43.590 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:43.591 INFO:teuthology.orchestra.run.smithi088.stdout:Total 17 MB/s | 91 MB 00:05 2024-05-15T12:20:43.591 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:20:43.596 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:20:43.596 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:20:44.188 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:20:44.189 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:20:45.307 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): ceph-debuginfo-19.0.0-1882.gae248125.el9 5.1 MB/s | 23 MB 00:04 2024-05-15T12:20:47.334 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:20:54.530 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/2 2024-05-15T12:20:54.617 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 2/2 2024-05-15T12:20:55.197 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): ceph-base-debuginfo-19.0.0-1882.gae24812 4.8 MB/s | 68 MB 00:14 2024-05-15T12:20:55.199 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:20:55.200 INFO:teuthology.orchestra.run.smithi049.stdout:Total 6.4 MB/s | 91 MB 00:14 2024-05-15T12:20:55.200 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:20:55.205 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:20:55.205 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:20:55.825 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:20:55.826 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:20:57.919 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 2/2 2024-05-15T12:20:57.920 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/2 2024-05-15T12:20:58.143 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:20:58.143 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:58.143 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:20:58.143 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:58.144 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:20:58.144 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:58.144 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:20:58.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-common-debuginfo 2024-05-15T12:20:58.965 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:44 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:20:59.088 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:20:59.088 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:59.088 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:20:59.088 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:59.088 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:20:59.088 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 422 M 2024-05-15T12:20:59.089 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:59.089 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:20:59.089 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:20:59.089 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:20:59.089 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:20:59.089 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 422 M 2024-05-15T12:20:59.089 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.7 G 2024-05-15T12:20:59.090 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:20:59.151 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:21:06.709 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/2 2024-05-15T12:21:06.793 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 2/2 2024-05-15T12:21:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 2/2 2024-05-15T12:21:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/2 2024-05-15T12:21:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2/2 2024-05-15T12:21:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:21:10.453 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:21:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:21:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:21:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:21:10.453 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:21:10.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-common-debuginfo 2024-05-15T12:21:11.287 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:59 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:21:11.419 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 422 M 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 422 M 2024-05-15T12:21:11.420 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.7 G 2024-05-15T12:21:11.421 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:21:57.912 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-common-debuginfo-19.0.0-1882.gae248125.el9 7.2 MB/s | 422 MB 00:58 2024-05-15T12:21:57.912 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:21:57.913 INFO:teuthology.orchestra.run.smithi088.stdout:Total 7.2 MB/s | 422 MB 00:58 2024-05-15T12:21:57.913 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:21:57.917 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:21:57.917 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:22:00.420 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:22:00.421 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:22:40.358 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-common-debuginfo-19.0.0-1882.gae248125.el9 4.7 MB/s | 422 MB 01:28 2024-05-15T12:22:40.359 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:22:40.359 INFO:teuthology.orchestra.run.smithi049.stdout:Total 4.7 MB/s | 422 MB 01:28 2024-05-15T12:22:40.360 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:22:40.365 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:22:40.365 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:22:42.999 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:22:43.000 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:22:43.171 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:22:43.286 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:22:51.265 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:22:51.491 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:22:51.492 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:22:51.492 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:22:51.492 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:22:51.492 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:22:51.492 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:22:51.736 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-05-15T12:22:52.328 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:04:38 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:22:52.458 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.4 M 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.4 M 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 6.5 M 2024-05-15T12:22:52.459 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:22:52.776 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-18 4.5 MB/s | 1.4 MB 00:00 2024-05-15T12:22:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:22:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:Total 4.5 MB/s | 1.4 MB 00:00 2024-05-15T12:22:52.778 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:22:52.784 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:22:52.784 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:22:52.847 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:22:52.847 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:22:53.124 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:22:53.224 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:22:53.616 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:22:53.841 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:22:53.841 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:22:53.841 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:22:53.841 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:22:53.841 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:22:53.842 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:22:54.001 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw-debuginfo 2024-05-15T12:22:54.577 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:04:40 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:22:54.700 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 176 M 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 176 M 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 757 M 2024-05-15T12:22:54.701 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:23:04.797 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-radosgw-debuginfo-19.0.0-1882.gae248125.el 17 MB/s | 176 MB 00:10 2024-05-15T12:23:04.797 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:23:04.798 INFO:teuthology.orchestra.run.smithi088.stdout:Total 17 MB/s | 176 MB 00:10 2024-05-15T12:23:04.798 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:23:04.802 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:23:04.802 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:23:05.863 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:23:05.864 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:23:23.933 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:23:24.016 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:23:28.333 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:23:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:23:28.634 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:23:28.859 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:23:28.860 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:23:28.860 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:23:28.860 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:23:28.860 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:23:28.860 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:23:29.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-test-debuginfo 2024-05-15T12:23:29.624 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:05:15 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:23:29.747 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:23:29.748 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:23:29.748 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:23:29.748 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:23:29.748 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:23:29.748 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 675 M 2024-05-15T12:23:29.748 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:23:29.748 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:23:29.748 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:23:29.749 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:23:29.749 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:23:29.749 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 675 M 2024-05-15T12:23:29.749 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 3.1 G 2024-05-15T12:23:29.749 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:23:32.063 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:23:32.280 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:23:32.280 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:23:32.280 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:23:32.280 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:23:32.280 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:23:32.280 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:23:32.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-05-15T12:23:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:05:21 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:23:33.270 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:23:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2024-05-15T12:23:33.271 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:23:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2024-05-15T12:23:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:23:33.271 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.4 M 2024-05-15T12:23:33.271 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:23:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:23:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2024-05-15T12:23:33.271 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:23:33.271 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:23:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 1.4 M 2024-05-15T12:23:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 6.5 M 2024-05-15T12:23:33.272 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:23:34.309 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-18 1.4 MB/s | 1.4 MB 00:01 2024-05-15T12:23:34.309 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:23:34.310 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.4 MB/s | 1.4 MB 00:01 2024-05-15T12:23:34.310 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:23:34.313 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:23:34.314 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:23:34.358 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:23:34.359 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:23:34.627 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:23:34.733 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:23:35.135 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:23:35.351 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1882. 1/1 2024-05-15T12:23:35.351 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:23:35.352 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:23:35.352 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:23:35.352 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:23:35.352 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:23:35.526 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw-debuginfo 2024-05-15T12:23:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:05:24 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:23:36.261 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:23:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:23:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:23:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:23:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:23:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 176 M 2024-05-15T12:23:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:23:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:23:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:23:36.262 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:23:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:23:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 176 M 2024-05-15T12:23:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 757 M 2024-05-15T12:23:36.263 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:23:58.020 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-radosgw-debuginfo-19.0.0-1882.gae248125.el 8.1 MB/s | 176 MB 00:21 2024-05-15T12:23:58.021 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:23:58.021 INFO:teuthology.orchestra.run.smithi049.stdout:Total 8.1 MB/s | 176 MB 00:21 2024-05-15T12:23:58.022 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:23:58.026 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:23:58.026 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:23:59.145 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:23:59.146 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:24:18.348 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:24:18.444 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:24:23.070 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:24:23.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x 1/1 2024-05-15T12:24:23.292 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:24:23.293 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:24:23.293 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:24:23.293 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:24:23.293 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:24:23.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test-debuginfo 2024-05-15T12:24:24.099 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:06:12 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:24:24.228 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:24:24.229 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:24:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:24:24.229 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:24:24.229 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:24:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 675 M 2024-05-15T12:24:24.229 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:24:24.230 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:24:24.230 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:24:24.230 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:24:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:24:24.230 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 675 M 2024-05-15T12:24:24.230 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 3.1 G 2024-05-15T12:24:24.230 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:24:25.785 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-test-debuginfo-19.0.0-1882.gae248125.el9.x 12 MB/s | 675 MB 00:56 2024-05-15T12:24:25.786 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:24:25.786 INFO:teuthology.orchestra.run.smithi088.stdout:Total 12 MB/s | 675 MB 00:56 2024-05-15T12:24:25.786 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:24:25.792 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:24:25.792 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:24:29.771 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:24:29.772 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:25:40.981 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:25:41.575 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:25:50.389 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:25:50.644 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:25:50.644 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:25:50.644 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:25:50.644 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:25:50.644 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:25:50.644 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:25:50.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base-debuginfo 2024-05-15T12:25:51.496 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:07:37 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:25:51.576 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:25:51.618 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:25:51.619 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-05-15T12:25:51.619 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:25:51.698 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-debuginfo 2024-05-15T12:25:52.276 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:07:38 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:25:52.399 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:25:52.399 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:25:52.399 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:25:52.400 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:25:52.400 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:25:52.400 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 20 M 2024-05-15T12:25:52.400 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:25:52.400 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:25:52.400 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:25:52.400 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:25:52.400 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:25:52.401 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 20 M 2024-05-15T12:25:52.401 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 104 M 2024-05-15T12:25:52.401 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:25:55.068 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-mgr-debuginfo-19.0.0-1882.gae248125.el9.x8 7.5 MB/s | 20 MB 00:02 2024-05-15T12:25:55.068 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:25:55.069 INFO:teuthology.orchestra.run.smithi088.stdout:Total 7.5 MB/s | 20 MB 00:02 2024-05-15T12:25:55.069 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:25:55.073 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:25:55.073 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:25:55.226 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:25:57.491 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:25:57.615 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:25:58.580 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:25:58.817 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:25:58.818 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:25:58.818 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:25:58.818 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:25:58.818 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:25:58.818 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:25:58.977 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mds-debuginfo 2024-05-15T12:25:59.551 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:07:45 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:25:59.675 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:25:59.675 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:25:59.675 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:25:59.675 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:25:59.675 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:25:59.676 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 49 M 2024-05-15T12:25:59.676 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:25:59.676 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:25:59.676 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:25:59.676 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:25:59.676 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:25:59.676 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 49 M 2024-05-15T12:25:59.677 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 177 M 2024-05-15T12:25:59.677 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:26:08.754 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-mds-debuginfo-19.0.0-1882.gae248125.el9.x8 5.4 MB/s | 49 MB 00:09 2024-05-15T12:26:08.755 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:26:08.755 INFO:teuthology.orchestra.run.smithi088.stdout:Total 5.3 MB/s | 49 MB 00:09 2024-05-15T12:26:08.755 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:26:08.759 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:26:08.760 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:26:09.082 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:26:09.082 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:26:13.665 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-test-debuginfo-19.0.0-1882.gae248125.el9.x 6.2 MB/s | 675 MB 01:49 2024-05-15T12:26:13.666 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:26:13.666 INFO:teuthology.orchestra.run.smithi049.stdout:Total 6.2 MB/s | 675 MB 01:49 2024-05-15T12:26:13.667 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:26:13.673 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:26:13.673 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:26:13.908 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:26:14.000 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:26:15.352 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:26:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:26:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:26:15.597 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:26:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:26:15.597 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:26:15.597 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:26:15.788 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mon-debuginfo 2024-05-15T12:26:16.364 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:02 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:26:16.487 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:26:16.488 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:26:16.488 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:26:16.488 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:26:16.488 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:26:16.488 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 99 M 2024-05-15T12:26:16.488 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:26:16.488 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:26:16.488 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:26:16.488 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:26:16.489 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:26:16.489 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 99 M 2024-05-15T12:26:16.489 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 450 M 2024-05-15T12:26:16.489 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:26:17.852 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:26:17.852 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:26:22.030 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-mon-debuginfo-19.0.0-1882.gae248125.el9.x8 18 MB/s | 99 MB 00:05 2024-05-15T12:26:22.031 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:26:22.031 INFO:teuthology.orchestra.run.smithi088.stdout:Total 18 MB/s | 99 MB 00:05 2024-05-15T12:26:22.032 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:26:22.035 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:26:22.036 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:26:22.648 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:26:22.648 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:26:32.928 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:26:33.026 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:26:35.839 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:26:36.099 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:26:36.099 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:26:36.099 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:26:36.099 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:26:36.099 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:26:36.099 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:26:36.275 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-osd-debuginfo 2024-05-15T12:26:36.851 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:22 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:26:36.973 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:26:36.974 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:26:36.974 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:26:36.974 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:26:36.974 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:26:36.974 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 289 M 2024-05-15T12:26:36.974 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:26:36.975 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:26:36.975 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:26:36.975 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:26:36.975 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:26:36.975 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 289 M 2024-05-15T12:26:36.975 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.2 G 2024-05-15T12:26:36.975 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:26:50.596 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-osd-debuginfo-19.0.0-1882.gae248125.el9.x8 21 MB/s | 289 MB 00:13 2024-05-15T12:26:50.597 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:26:50.598 INFO:teuthology.orchestra.run.smithi088.stdout:Total 21 MB/s | 289 MB 00:13 2024-05-15T12:26:50.598 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:26:50.602 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:26:50.602 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:26:52.318 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:26:52.318 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:27:21.906 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:27:21.998 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:27:29.585 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:27:29.820 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:27:29.820 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:29.820 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:27:29.821 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:27:29.821 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:29.821 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:27:30.038 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-fuse-debuginfo 2024-05-15T12:27:30.616 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:09:16 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:27:30.738 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:27:30.739 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:30.739 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:27:30.739 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:30.739 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:27:30.739 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 6.0 M 2024-05-15T12:27:30.739 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:30.740 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:27:30.740 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:30.740 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:27:30.740 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:30.740 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 6.0 M 2024-05-15T12:27:30.740 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 28 M 2024-05-15T12:27:30.740 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:27:32.053 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-fuse-debuginfo-19.0.0-1882.gae248125.el9.x 4.6 MB/s | 6.0 MB 00:01 2024-05-15T12:27:32.053 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:27:32.054 INFO:teuthology.orchestra.run.smithi088.stdout:Total 4.5 MB/s | 6.0 MB 00:01 2024-05-15T12:27:32.054 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:27:32.057 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:27:32.058 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:27:32.127 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:27:32.128 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:27:32.873 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:27:32.976 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:27:33.460 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:27:33.495 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:27:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:27:33.738 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:27:33.738 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:33.738 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:27:33.738 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:27:33.738 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:33.738 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:27:33.894 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados-devel-debuginfo 2024-05-15T12:27:34.468 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:09:20 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:27:34.593 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:27:34.593 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:34.593 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 109 k 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 109 k 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 440 k 2024-05-15T12:27:34.594 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:27:35.005 INFO:teuthology.orchestra.run.smithi088.stdout:librados-devel-debuginfo-19.0.0-1882.gae248125. 265 kB/s | 109 kB 00:00 2024-05-15T12:27:35.006 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:27:35.006 INFO:teuthology.orchestra.run.smithi088.stdout:Total 264 kB/s | 109 kB 00:00 2024-05-15T12:27:35.006 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:27:35.010 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:27:35.010 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:27:35.046 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:27:35.047 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:27:35.169 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:27:35.277 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:27:35.611 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:27:35.844 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:27:35.845 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:35.845 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:27:35.845 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:27:35.845 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:35.845 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:27:36.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs2-debuginfo 2024-05-15T12:27:36.596 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:09:22 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:27:36.718 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:27:36.719 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:36.719 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:27:36.719 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:36.719 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:27:36.719 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.4 M 2024-05-15T12:27:36.719 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:36.719 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:27:36.719 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:36.719 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:27:36.719 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:36.720 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 5.4 M 2024-05-15T12:27:36.720 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 25 M 2024-05-15T12:27:36.720 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:27:40.057 INFO:teuthology.orchestra.run.smithi088.stdout:libcephfs2-debuginfo-19.0.0-1882.gae248125.el9. 1.6 MB/s | 5.4 MB 00:03 2024-05-15T12:27:40.058 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:27:40.058 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.6 MB/s | 5.4 MB 00:03 2024-05-15T12:27:40.059 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:27:40.063 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:27:40.063 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:27:40.132 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:27:40.132 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:27:40.794 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:27:40.905 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:27:41.177 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:27:41.389 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:27:41.417 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:27:41.417 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:27:41.417 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:27:41.417 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:27:41.417 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:27:41.417 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:27:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:27:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:41.618 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:27:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:27:41.618 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:41.618 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:27:41.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base-debuginfo 2024-05-15T12:27:41.796 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados2-debuginfo 2024-05-15T12:27:42.302 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:09:30 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:27:42.375 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:09:28 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:27:42.385 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 is already installed. 2024-05-15T12:27:42.430 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:27:42.431 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-05-15T12:27:42.431 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:27:42.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-debuginfo 2024-05-15T12:27:42.497 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:27:42.498 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:42.498 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:27:42.498 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:42.498 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:27:42.498 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 45 M 2024-05-15T12:27:42.498 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:42.498 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:27:42.498 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:42.498 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:27:42.498 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:42.499 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 45 M 2024-05-15T12:27:42.499 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 255 M 2024-05-15T12:27:42.499 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:27:43.091 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:09:31 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:27:43.219 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:27:43.220 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:27:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:27:43.220 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:27:43.220 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:27:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 20 M 2024-05-15T12:27:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:27:43.221 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:27:43.221 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:27:43.221 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:27:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:27:43.221 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 20 M 2024-05-15T12:27:43.221 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 104 M 2024-05-15T12:27:43.221 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:27:46.793 INFO:teuthology.orchestra.run.smithi088.stdout:librados2-debuginfo-19.0.0-1882.gae248125.el9.x 10 MB/s | 45 MB 00:04 2024-05-15T12:27:46.794 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:27:46.794 INFO:teuthology.orchestra.run.smithi088.stdout:Total 10 MB/s | 45 MB 00:04 2024-05-15T12:27:46.795 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:27:46.799 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:27:46.799 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:27:47.101 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:27:47.101 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:27:47.289 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-mgr-debuginfo-19.0.0-1882.gae248125.el9.x8 4.9 MB/s | 20 MB 00:04 2024-05-15T12:27:47.289 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:27:47.290 INFO:teuthology.orchestra.run.smithi049.stdout:Total 4.9 MB/s | 20 MB 00:04 2024-05-15T12:27:47.290 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:27:47.294 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:27:47.295 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:27:47.457 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:27:47.458 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:27:49.860 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:27:49.973 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:27:50.952 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:27:51.175 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:27:51.175 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:27:51.176 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:27:51.176 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:27:51.176 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:27:51.176 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:27:51.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mds-debuginfo 2024-05-15T12:27:51.958 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:09:39 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:27:52.084 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:27:52.085 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:27:52.085 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:27:52.085 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:27:52.086 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:27:52.086 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 49 M 2024-05-15T12:27:52.086 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:27:52.086 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:27:52.086 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:27:52.086 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:27:52.086 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:27:52.086 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 49 M 2024-05-15T12:27:52.086 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 177 M 2024-05-15T12:27:52.086 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:27:52.209 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:27:52.307 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:27:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:27:54.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:27:54.415 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:54.415 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:27:54.415 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:27:54.415 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:54.415 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:27:54.606 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librbd1-debuginfo 2024-05-15T12:27:55.181 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:09:41 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:27:55.304 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:27:55.305 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:27:55.305 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:55.305 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:27:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 52 M 2024-05-15T12:27:55.305 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:55.306 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:27:55.306 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:27:55.306 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:27:55.306 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:27:55.306 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 52 M 2024-05-15T12:27:55.306 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 308 M 2024-05-15T12:27:55.306 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:27:59.274 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-mds-debuginfo-19.0.0-1882.gae248125.el9.x8 6.8 MB/s | 49 MB 00:07 2024-05-15T12:27:59.275 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:27:59.275 INFO:teuthology.orchestra.run.smithi049.stdout:Total 6.8 MB/s | 49 MB 00:07 2024-05-15T12:27:59.275 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:27:59.280 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:27:59.280 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:27:59.616 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:27:59.616 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:28:00.869 INFO:teuthology.orchestra.run.smithi088.stdout:librbd1-debuginfo-19.0.0-1882.gae248125.el9.x86 9.3 MB/s | 52 MB 00:05 2024-05-15T12:28:00.870 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:28:00.871 INFO:teuthology.orchestra.run.smithi088.stdout:Total 9.3 MB/s | 52 MB 00:05 2024-05-15T12:28:00.871 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:28:00.875 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:28:00.875 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:28:01.221 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:28:01.222 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:28:04.754 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:28:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:28:06.191 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:28:06.420 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:28:06.420 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:28:06.421 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:28:06.421 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:28:06.421 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:28:06.421 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:28:06.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mon-debuginfo 2024-05-15T12:28:07.218 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:09:55 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:28:07.254 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:28:07.349 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:28:07.350 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:28:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:28:07.350 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:28:07.351 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:28:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 99 M 2024-05-15T12:28:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:28:07.351 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:28:07.351 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:28:07.351 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:28:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:28:07.351 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 99 M 2024-05-15T12:28:07.351 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 450 M 2024-05-15T12:28:07.351 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:28:07.432 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:28:09.587 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:28:09.828 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:28:09.828 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:09.828 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:28:09.828 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:28:09.828 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:09.828 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:28:10.018 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-cephfs-debuginfo 2024-05-15T12:28:10.594 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:09:56 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:28:10.716 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:28:10.717 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:10.717 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:28:10.717 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:10.717 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:28:10.717 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 615 k 2024-05-15T12:28:10.717 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:10.717 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:28:10.718 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:10.718 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:28:10.718 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:10.718 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 615 k 2024-05-15T12:28:10.718 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.1 M 2024-05-15T12:28:10.718 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:28:12.246 INFO:teuthology.orchestra.run.smithi088.stdout:python3-cephfs-debuginfo-19.0.0-1882.gae248125. 402 kB/s | 615 kB 00:01 2024-05-15T12:28:12.246 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:28:12.247 INFO:teuthology.orchestra.run.smithi088.stdout:Total 402 kB/s | 615 kB 00:01 2024-05-15T12:28:12.247 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:28:12.251 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:28:12.251 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:28:12.295 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:28:12.295 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:28:12.483 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:28:12.578 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:28:12.964 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:28:13.203 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:28:13.203 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:13.204 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:28:13.204 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:28:13.204 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:13.204 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:28:13.388 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rados-debuginfo 2024-05-15T12:28:13.967 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:09:59 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:28:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:28:14.089 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 769 k 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 769 k 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.7 M 2024-05-15T12:28:14.090 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:28:14.690 INFO:teuthology.orchestra.run.smithi088.stdout:python3-rados-debuginfo-19.0.0-1882.gae248125.e 1.3 MB/s | 769 kB 00:00 2024-05-15T12:28:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:28:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.2 MB/s | 769 kB 00:00 2024-05-15T12:28:14.691 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:28:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:28:14.695 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:28:14.733 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:28:14.733 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:28:14.912 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:28:15.014 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:28:15.338 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:28:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:28:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:15.578 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:28:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:28:15.578 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:15.578 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:28:15.763 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rbd-debuginfo 2024-05-15T12:28:16.338 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:10:02 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:28:16.462 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:28:16.463 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:16.463 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:28:16.463 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:16.463 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:28:16.463 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.1 M 2024-05-15T12:28:16.463 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:16.463 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:28:16.463 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:16.464 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:28:16.464 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:16.464 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.1 M 2024-05-15T12:28:16.464 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 4.3 M 2024-05-15T12:28:16.464 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:28:17.143 INFO:teuthology.orchestra.run.smithi088.stdout:python3-rbd-debuginfo-19.0.0-1882.gae248125.el9 1.7 MB/s | 1.1 MB 00:00 2024-05-15T12:28:17.144 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:28:17.144 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.7 MB/s | 1.1 MB 00:00 2024-05-15T12:28:17.144 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:28:17.148 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:28:17.148 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:28:17.191 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:28:17.191 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:28:17.433 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:28:17.524 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:28:17.913 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:28:18.151 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:28:18.151 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:18.151 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:28:18.151 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:28:18.151 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:18.152 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:28:18.342 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rgw-debuginfo 2024-05-15T12:28:18.917 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:10:04 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 337 k 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:28:19.039 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:19.040 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 337 k 2024-05-15T12:28:19.040 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.0 M 2024-05-15T12:28:19.040 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:28:19.588 INFO:teuthology.orchestra.run.smithi088.stdout:python3-rgw-debuginfo-19.0.0-1882.gae248125.el9 616 kB/s | 337 kB 00:00 2024-05-15T12:28:19.588 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:28:19.588 INFO:teuthology.orchestra.run.smithi088.stdout:Total 615 kB/s | 337 kB 00:00 2024-05-15T12:28:19.589 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:28:19.592 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:28:19.592 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:28:19.629 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:28:19.629 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:28:19.767 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:28:19.869 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:28:20.217 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:28:20.470 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:28:20.470 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:20.470 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:28:20.470 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:28:20.470 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:20.470 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:28:20.647 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-fuse-debuginfo 2024-05-15T12:28:21.221 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:10:07 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:28:21.344 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 517 k 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 517 k 2024-05-15T12:28:21.345 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.6 M 2024-05-15T12:28:21.346 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:28:22.049 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-fuse-debuginfo-19.0.0-1882.gae248125.el9.x8 734 kB/s | 517 kB 00:00 2024-05-15T12:28:22.050 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:28:22.050 INFO:teuthology.orchestra.run.smithi088.stdout:Total 733 kB/s | 517 kB 00:00 2024-05-15T12:28:22.050 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:28:22.053 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:28:22.054 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:28:22.093 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:28:22.093 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:28:22.257 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:28:22.355 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:28:22.739 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:28:22.966 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:28:22.967 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:22.967 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:28:22.967 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:28:22.967 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:22.967 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:28:23.140 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-mirror-debuginfo 2024-05-15T12:28:23.717 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:10:09 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:28:23.841 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:28:23.841 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:23.841 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 52 M 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 52 M 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 298 M 2024-05-15T12:28:23.842 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:28:36.866 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-mon-debuginfo-19.0.0-1882.gae248125.el9.x8 3.3 MB/s | 99 MB 00:29 2024-05-15T12:28:36.867 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:28:36.867 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.3 MB/s | 99 MB 00:29 2024-05-15T12:28:36.868 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:28:36.872 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:28:36.872 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:28:36.925 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-mirror-debuginfo-19.0.0-1882.gae248125.el9. 3.9 MB/s | 52 MB 00:13 2024-05-15T12:28:36.926 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:28:36.926 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.9 MB/s | 52 MB 00:13 2024-05-15T12:28:36.926 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:28:36.930 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:28:36.930 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:28:37.269 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:28:37.269 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:28:37.516 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:28:37.517 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:28:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:28:43.288 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:28:45.383 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:28:45.621 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:28:45.621 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:45.621 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:28:45.621 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:28:45.621 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:45.622 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:28:45.780 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-nbd-debuginfo 2024-05-15T12:28:46.357 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:10:32 ago on Wed 15 May 2024 12:18:14 PM UTC. 2024-05-15T12:28:46.482 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.3 M 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.3 M 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 5.7 M 2024-05-15T12:28:46.483 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-15T12:28:47.084 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-nbd-debuginfo-19.0.0-1882.gae248125.el9.x86 2.1 MB/s | 1.3 MB 00:00 2024-05-15T12:28:47.085 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:28:47.085 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.1 MB/s | 1.3 MB 00:00 2024-05-15T12:28:47.085 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-15T12:28:47.089 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-15T12:28:47.089 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-15T12:28:47.133 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-15T12:28:47.133 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-15T12:28:47.395 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-15T12:28:47.499 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:28:47.931 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:28:48.170 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:28:48.170 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:48.170 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-15T12:28:48.170 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:28:48.170 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:28:48.170 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-15T12:28:48.335 DEBUG:teuthology.parallel:result is None 2024-05-15T12:28:48.454 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:28:48.557 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:28:51.456 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:28:51.696 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:28:51.696 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:28:51.696 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:28:51.696 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:28:51.696 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:28:51.696 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:28:51.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-osd-debuginfo 2024-05-15T12:28:52.453 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:10:40 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:28:52.581 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:28:52.581 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:28:52.581 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:28:52.581 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:28:52.582 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:28:52.582 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 289 M 2024-05-15T12:28:52.582 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:28:52.582 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:28:52.582 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:28:52.582 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:28:52.582 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:28:52.582 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 289 M 2024-05-15T12:28:52.582 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.2 G 2024-05-15T12:28:52.582 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:29:12.716 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-osd-debuginfo-19.0.0-1882.gae248125.el9.x8 14 MB/s | 289 MB 00:20 2024-05-15T12:29:12.716 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:29:12.717 INFO:teuthology.orchestra.run.smithi049.stdout:Total 14 MB/s | 289 MB 00:20 2024-05-15T12:29:12.717 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:29:12.721 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:29:12.721 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:29:14.526 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:29:14.526 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:29:45.875 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:29:45.982 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:29:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:29:53.871 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:29:53.871 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:29:53.872 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:29:53.872 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:29:53.872 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:29:53.872 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:29:54.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse-debuginfo 2024-05-15T12:29:54.681 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:11:42 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:29:54.809 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:29:54.810 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:29:54.810 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:29:54.810 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:29:54.811 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:29:54.811 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 6.0 M 2024-05-15T12:29:54.811 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:29:54.811 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:29:54.811 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:29:54.811 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:29:54.811 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:29:54.811 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 6.0 M 2024-05-15T12:29:54.811 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 28 M 2024-05-15T12:29:54.811 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:29:55.492 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-fuse-debuginfo-19.0.0-1882.gae248125.el9.x 8.8 MB/s | 6.0 MB 00:00 2024-05-15T12:29:55.492 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:29:55.493 INFO:teuthology.orchestra.run.smithi049.stdout:Total 8.8 MB/s | 6.0 MB 00:00 2024-05-15T12:29:55.493 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:29:55.496 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:29:55.497 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:29:55.571 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:29:55.571 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:29:56.346 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:29:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:29:56.939 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:29:57.154 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:29:57.154 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:29:57.154 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:29:57.154 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:29:57.154 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:29:57.155 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:29:57.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel-debuginfo 2024-05-15T12:29:57.917 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:11:45 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:29:58.046 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:29:58.047 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:29:58.047 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:29:58.047 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:29:58.047 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:29:58.048 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 109 k 2024-05-15T12:29:58.048 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:29:58.048 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:29:58.048 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:29:58.048 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:29:58.048 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:29:58.048 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 109 k 2024-05-15T12:29:58.048 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 440 k 2024-05-15T12:29:58.048 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:29:58.695 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-debuginfo-19.0.0-1882.gae248125. 168 kB/s | 109 kB 00:00 2024-05-15T12:29:58.695 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:29:58.696 INFO:teuthology.orchestra.run.smithi049.stdout:Total 168 kB/s | 109 kB 00:00 2024-05-15T12:29:58.696 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:29:58.699 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:29:58.699 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:29:58.739 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:29:58.739 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:29:58.861 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:29:58.955 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:29:59.297 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:29:59.529 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:29:59.529 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:29:59.529 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:29:59.529 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:29:59.529 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:29:59.530 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:29:59.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2-debuginfo 2024-05-15T12:30:00.296 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:11:48 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:30:00.425 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:30:00.426 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:30:00.426 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:00.426 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:30:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 5.4 M 2024-05-15T12:30:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:00.427 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:30:00.427 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:00.427 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:30:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:00.427 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 5.4 M 2024-05-15T12:30:00.427 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 25 M 2024-05-15T12:30:00.427 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:30:01.056 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs2-debuginfo-19.0.0-1882.gae248125.el9. 8.6 MB/s | 5.4 MB 00:00 2024-05-15T12:30:01.056 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:30:01.057 INFO:teuthology.orchestra.run.smithi049.stdout:Total 8.5 MB/s | 5.4 MB 00:00 2024-05-15T12:30:01.057 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:30:01.061 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:30:01.061 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:30:01.132 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:30:01.132 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:30:01.840 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:30:01.974 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:30:02.475 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:30:02.703 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:30:02.704 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:02.704 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:30:02.704 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:30:02.704 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:02.704 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:30:02.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2-debuginfo 2024-05-15T12:30:03.499 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:11:51 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:30:03.628 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:30:03.628 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 45 M 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 45 M 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 255 M 2024-05-15T12:30:03.629 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:30:05.825 INFO:teuthology.orchestra.run.smithi049.stdout:librados2-debuginfo-19.0.0-1882.gae248125.el9.x 20 MB/s | 45 MB 00:02 2024-05-15T12:30:05.826 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:30:05.827 INFO:teuthology.orchestra.run.smithi049.stdout:Total 20 MB/s | 45 MB 00:02 2024-05-15T12:30:05.827 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:30:05.832 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:30:05.832 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:30:06.150 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:30:06.150 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:30:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:30:11.660 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:30:13.474 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:30:13.692 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_ 1/1 2024-05-15T12:30:13.692 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:13.692 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:30:13.692 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:30:13.692 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:13.692 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:30:13.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1-debuginfo 2024-05-15T12:30:14.476 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:12:02 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:30:14.605 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 52 M 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:14.606 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 52 M 2024-05-15T12:30:14.607 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 308 M 2024-05-15T12:30:14.607 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:30:18.927 INFO:teuthology.orchestra.run.smithi049.stdout:librbd1-debuginfo-19.0.0-1882.gae248125.el9.x86 12 MB/s | 52 MB 00:04 2024-05-15T12:30:18.928 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:30:18.929 INFO:teuthology.orchestra.run.smithi049.stdout:Total 12 MB/s | 52 MB 00:04 2024-05-15T12:30:18.929 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:30:18.933 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:30:18.933 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:30:19.295 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:30:19.295 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:30:25.668 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:30:25.769 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:30:27.916 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:30:28.139 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:30:28.140 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:28.140 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:30:28.140 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:30:28.140 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:28.140 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:30:28.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs-debuginfo 2024-05-15T12:30:28.934 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:12:16 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:30:29.064 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:30:29.065 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:30:29.065 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:29.065 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:30:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 615 k 2024-05-15T12:30:29.065 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:29.065 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:30:29.065 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:29.065 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:30:29.066 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:29.066 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 615 k 2024-05-15T12:30:29.066 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 2.1 M 2024-05-15T12:30:29.066 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:30:29.372 INFO:teuthology.orchestra.run.smithi049.stdout:python3-cephfs-debuginfo-19.0.0-1882.gae248125. 2.0 MB/s | 615 kB 00:00 2024-05-15T12:30:29.373 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:30:29.373 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.0 MB/s | 615 kB 00:00 2024-05-15T12:30:29.373 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:30:29.376 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:30:29.376 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:30:29.419 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:30:29.419 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:30:29.591 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:30:29.698 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:30:30.061 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:30:30.284 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9 1/1 2024-05-15T12:30:30.284 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:30.285 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:30:30.285 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:30:30.285 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:30.285 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:30:30.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados-debuginfo 2024-05-15T12:30:31.066 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:12:19 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:30:31.194 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:30:31.195 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:31.195 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:30:31.195 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:31.195 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:30:31.195 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 769 k 2024-05-15T12:30:31.195 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:31.195 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:30:31.196 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:31.196 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:30:31.196 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:31.196 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 769 k 2024-05-15T12:30:31.196 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 2.7 M 2024-05-15T12:30:31.196 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:30:32.116 INFO:teuthology.orchestra.run.smithi049.stdout:python3-rados-debuginfo-19.0.0-1882.gae248125.e 836 kB/s | 769 kB 00:00 2024-05-15T12:30:32.117 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:30:32.118 INFO:teuthology.orchestra.run.smithi049.stdout:Total 835 kB/s | 769 kB 00:00 2024-05-15T12:30:32.118 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:30:32.122 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:30:32.122 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:30:32.168 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:30:32.168 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:30:32.368 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:30:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:30:32.849 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:30:33.084 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9. 1/1 2024-05-15T12:30:33.084 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:30:33.084 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:30:33.084 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:33.084 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:30:33.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd-debuginfo 2024-05-15T12:30:33.850 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:12:21 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:30:33.978 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:30:33.979 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:33.979 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:30:33.979 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:33.979 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:30:33.979 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.1 M 2024-05-15T12:30:33.979 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:33.979 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:30:33.979 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:33.979 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:30:33.979 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:33.980 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 1.1 M 2024-05-15T12:30:33.980 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 4.3 M 2024-05-15T12:30:33.980 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:30:34.495 INFO:teuthology.orchestra.run.smithi049.stdout:python3-rbd-debuginfo-19.0.0-1882.gae248125.el9 2.2 MB/s | 1.1 MB 00:00 2024-05-15T12:30:34.495 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:30:34.495 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.2 MB/s | 1.1 MB 00:00 2024-05-15T12:30:34.495 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:30:34.499 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:30:34.499 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:30:34.545 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:30:34.545 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:30:34.781 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:30:34.896 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:30:35.264 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:30:35.492 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:30:35.492 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:35.492 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:30:35.492 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:30:35.492 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:35.492 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:30:35.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw-debuginfo 2024-05-15T12:30:36.278 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:12:24 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:30:36.407 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:30:36.407 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:36.407 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 337 k 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 337 k 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.0 M 2024-05-15T12:30:36.408 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:30:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:python3-rgw-debuginfo-19.0.0-1882.gae248125.el9 799 kB/s | 337 kB 00:00 2024-05-15T12:30:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:30:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:Total 796 kB/s | 337 kB 00:00 2024-05-15T12:30:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:30:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:30:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:30:36.876 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:30:36.877 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:30:37.019 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:30:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:30:37.528 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:30:37.766 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x8 1/1 2024-05-15T12:30:37.766 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:37.766 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:30:37.766 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:30:37.766 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:37.766 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:30:37.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse-debuginfo 2024-05-15T12:30:38.551 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:12:26 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:30:38.679 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:30:38.681 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:38.681 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:30:38.681 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:38.681 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:30:38.681 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 517 k 2024-05-15T12:30:38.681 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:38.681 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:30:38.681 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:38.681 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:30:38.681 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:38.682 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 517 k 2024-05-15T12:30:38.682 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 2.6 M 2024-05-15T12:30:38.682 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:30:38.993 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-debuginfo-19.0.0-1882.gae248125.el9.x8 1.6 MB/s | 517 kB 00:00 2024-05-15T12:30:38.993 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:30:38.993 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.6 MB/s | 517 kB 00:00 2024-05-15T12:30:38.994 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:30:38.997 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:30:38.997 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:30:39.037 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:30:39.037 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:30:39.210 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:30:39.318 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:30:39.665 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:30:39.888 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_6 1/1 2024-05-15T12:30:39.888 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:39.888 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:30:39.888 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:30:39.888 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:39.888 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:30:40.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror-debuginfo 2024-05-15T12:30:40.659 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:12:28 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:30:40.788 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:30:40.788 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 52 M 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 52 M 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 298 M 2024-05-15T12:30:40.789 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:30:44.820 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-debuginfo-19.0.0-1882.gae248125.el9. 13 MB/s | 52 MB 00:04 2024-05-15T12:30:44.821 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:30:44.822 INFO:teuthology.orchestra.run.smithi049.stdout:Total 13 MB/s | 52 MB 00:04 2024-05-15T12:30:44.822 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:30:44.826 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:30:44.826 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:30:45.181 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:30:45.181 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:30:51.445 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:30:51.545 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:30:53.576 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:30:53.811 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86 1/1 2024-05-15T12:30:53.811 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:53.811 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:30:53.811 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:30:53.811 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:53.811 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:30:53.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd-debuginfo 2024-05-15T12:30:54.568 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:12:42 ago on Wed 15 May 2024 12:18:12 PM UTC. 2024-05-15T12:30:54.697 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-05-15T12:30:54.698 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:54.698 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-05-15T12:30:54.698 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:54.699 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-05-15T12:30:54.699 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1882.gae248125.el9 ceph 1.3 M 2024-05-15T12:30:54.699 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:54.699 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-05-15T12:30:54.699 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-05-15T12:30:54.699 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-05-15T12:30:54.699 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:54.700 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 1.3 M 2024-05-15T12:30:54.700 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 5.7 M 2024-05-15T12:30:54.700 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-05-15T12:30:55.597 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-debuginfo-19.0.0-1882.gae248125.el9.x86 1.4 MB/s | 1.3 MB 00:00 2024-05-15T12:30:55.598 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:30:55.598 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.4 MB/s | 1.3 MB 00:00 2024-05-15T12:30:55.598 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-05-15T12:30:55.602 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-05-15T12:30:55.602 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-05-15T12:30:55.651 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-05-15T12:30:55.651 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-05-15T12:30:55.908 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-05-15T12:30:56.015 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:30:56.405 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:30:56.629 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 1/1 2024-05-15T12:30:56.630 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:56.630 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-05-15T12:30:56.630 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-debuginfo-2:19.0.0-1882.gae248125.el9.x86_64 2024-05-15T12:30:56.630 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:30:56.630 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-05-15T12:30:56.807 DEBUG:teuthology.parallel:result is None 2024-05-15T12:30:56.807 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:30:56.950 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-15T12:30:56.972 INFO:teuthology.orchestra.run.smithi049.stdout:19.0.0-1882.gae248125.el9 2024-05-15T12:30:56.972 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1882.gae248125.el9 2024-05-15T12:30:56.972 INFO:teuthology.task.install:The correct ceph version 19.0.0-1882.gae248125 is installed. 2024-05-15T12:30:56.974 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:30:57.113 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-15T12:30:57.139 INFO:teuthology.orchestra.run.smithi088.stdout:19.0.0-1882.gae248125.el9 2024-05-15T12:30:57.139 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1882.gae248125.el9 2024-05-15T12:30:57.139 INFO:teuthology.task.install:The correct ceph version 19.0.0-1882.gae248125 is installed. 2024-05-15T12:30:57.141 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-15T12:30:57.141 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:30:57.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T12:30:57.167 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:30:57.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-15T12:30:57.207 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-15T12:30:57.207 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:30:57.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T12:30:57.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T12:30:57.298 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:30:57.299 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-05-15T12:30:57.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-15T12:30:57.393 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-15T12:30:57.393 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:30:57.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T12:30:57.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T12:30:57.484 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:30:57.484 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-15T12:30:57.510 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-15T12:30:57.576 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-15T12:30:57.576 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:30:57.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T12:30:57.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T12:30:57.667 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:30:57.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-05-15T12:30:57.692 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-15T12:30:57.757 INFO:teuthology.run_tasks:Running task ceph... 2024-05-15T12:30:57.834 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-15T12:30:57.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /var/log/ceph 2024-05-15T12:30:57.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /var/log/ceph 2024-05-15T12:30:57.861 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-15T12:30:57.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-15T12:30:57.902 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-15T12:30:57.926 INFO:tasks.ceph:Creating extra log directories... 2024-05-15T12:30:57.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-15T12:30:57.969 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-15T12:30:58.004 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-15T12:30:58.004 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-15T12:30:58.004 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091', 'branch': 'wip-osd-ec-partial-reads', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7707091', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-osd-ec-partial-reads'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-osd-ec-partial-reads', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'}}, 'owner': 'scheduled_rzarzynski@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8148, 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-osd-ec-partial-reads', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'targets': {'smithi049.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEmLW9QW9ln5PsrW3X0qHTFkDw20rYYIOHorXizGP+J22ldFT2fYmnZS5JhVJgCj+5mKoAw8uR2ftY5DxXiwXI=', 'smithi088.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDHUNiwRS4lgXULAUrwjuTa4tocxkAsPXngfwENURgnL63jMvpSce2NjqdKwO3nce9ZCa2Zb7XDQ1qUVJKThM/Q='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-15_11:53:37', 'tube': 'smithi', 'user': 'rzarzynski', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-15T12:30:58.004 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-15T12:30:58.035 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-15T12:30:58.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/run/ceph 2024-05-15T12:30:58.090 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/run/ceph 2024-05-15T12:30:58.115 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:30:58.115 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:30:58.170 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T12:30:58.170 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2024-05-15T12:30:58.226 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T12:30:58.226 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:58.226 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-15T12:30:58.226 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:58.226 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:58.226 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-05-15 12:20:36.317077073 +0000 2024-05-15T12:30:58.226 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-05-15 12:17:10.246638669 +0000 2024-05-15T12:30:58.226 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-05-15 12:17:10.246638669 +0000 2024-05-15T12:30:58.227 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-05-15 12:17:10.246638669 +0000 2024-05-15T12:30:58.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T12:30:58.289 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-05-15T12:30:58.290 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-05-15T12:30:58.290 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000104113 s, 4.9 MB/s 2024-05-15T12:30:58.291 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T12:30:58.348 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2024-05-15T12:30:58.404 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T12:30:58.404 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:58.404 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-05-15T12:30:58.404 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:58.404 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:58.404 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-05-15 12:20:36.317077073 +0000 2024-05-15T12:30:58.405 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-05-15 12:17:10.588629115 +0000 2024-05-15T12:30:58.405 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-05-15 12:17:10.588629115 +0000 2024-05-15T12:30:58.405 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-05-15 12:17:10.588629115 +0000 2024-05-15T12:30:58.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T12:30:58.468 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-05-15T12:30:58.468 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-05-15T12:30:58.468 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 8.7521e-05 s, 5.9 MB/s 2024-05-15T12:30:58.469 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T12:30:58.527 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2024-05-15T12:30:58.582 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T12:30:58.582 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:58.582 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-15T12:30:58.583 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:58.583 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:58.583 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-05-15 12:20:36.317077073 +0000 2024-05-15T12:30:58.583 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-05-15 12:17:10.921619812 +0000 2024-05-15T12:30:58.583 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-05-15 12:17:10.921619812 +0000 2024-05-15T12:30:58.583 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-05-15 12:17:10.921619812 +0000 2024-05-15T12:30:58.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T12:30:58.647 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-05-15T12:30:58.647 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-05-15T12:30:58.647 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 9.3409e-05 s, 5.5 MB/s 2024-05-15T12:30:58.648 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T12:30:58.705 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2024-05-15T12:30:58.761 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T12:30:58.761 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:58.761 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-15T12:30:58.761 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:58.761 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:58.761 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-05-15 12:20:36.318077045 +0000 2024-05-15T12:30:58.761 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-05-15 12:17:11.247610705 +0000 2024-05-15T12:30:58.761 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-05-15 12:17:11.247610705 +0000 2024-05-15T12:30:58.761 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-05-15 12:17:11.247610705 +0000 2024-05-15T12:30:58.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T12:30:58.824 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-05-15T12:30:58.825 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-05-15T12:30:58.825 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000100939 s, 5.1 MB/s 2024-05-15T12:30:58.826 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T12:30:58.882 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-15T12:30:58.883 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:30:58.883 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:30:58.900 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T12:30:58.900 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2024-05-15T12:30:58.956 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T12:30:58.956 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:58.956 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-05-15T12:30:58.956 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:58.956 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:58.956 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-15 12:20:33.867823986 +0000 2024-05-15T12:30:58.957 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-15 12:17:03.743372901 +0000 2024-05-15T12:30:58.957 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-15 12:17:03.743372901 +0000 2024-05-15T12:30:58.957 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-05-15 12:17:03.743372901 +0000 2024-05-15T12:30:58.957 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T12:30:59.021 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-15T12:30:59.022 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-15T12:30:59.022 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000216349 s, 2.4 MB/s 2024-05-15T12:30:59.023 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T12:30:59.080 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2024-05-15T12:30:59.137 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T12:30:59.137 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:59.137 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-05-15T12:30:59.137 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:59.137 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:59.137 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-15 12:20:33.867823986 +0000 2024-05-15T12:30:59.137 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-15 12:17:04.080366188 +0000 2024-05-15T12:30:59.137 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-15 12:17:04.080366188 +0000 2024-05-15T12:30:59.138 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-05-15 12:17:04.080366188 +0000 2024-05-15T12:30:59.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T12:30:59.201 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-15T12:30:59.201 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-15T12:30:59.201 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000168588 s, 3.0 MB/s 2024-05-15T12:30:59.203 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T12:30:59.259 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2024-05-15T12:30:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T12:30:59.315 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:59.315 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-05-15T12:30:59.315 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:59.316 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:59.316 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-15 12:20:33.868823965 +0000 2024-05-15T12:30:59.316 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-15 12:17:04.408359653 +0000 2024-05-15T12:30:59.316 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-15 12:17:04.408359653 +0000 2024-05-15T12:30:59.316 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-05-15 12:17:04.408359653 +0000 2024-05-15T12:30:59.316 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T12:30:59.379 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-15T12:30:59.379 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-15T12:30:59.379 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000226668 s, 2.3 MB/s 2024-05-15T12:30:59.380 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T12:30:59.436 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2024-05-15T12:30:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T12:30:59.492 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:59.492 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-15T12:30:59.492 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:59.492 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:59.492 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-15 12:20:33.868823965 +0000 2024-05-15T12:30:59.493 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-15 12:17:04.739353059 +0000 2024-05-15T12:30:59.493 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-15 12:17:04.739353059 +0000 2024-05-15T12:30:59.493 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-05-15 12:17:04.739353059 +0000 2024-05-15T12:30:59.493 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T12:30:59.555 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-15T12:30:59.555 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-15T12:30:59.555 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000189439 s, 2.7 MB/s 2024-05-15T12:30:59.556 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T12:30:59.613 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-15T12:30:59.613 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-15T12:30:59.613 INFO:tasks.ceph:Generating config... 2024-05-15T12:30:59.614 INFO:tasks.ceph:[client] debug monc = 20 2024-05-15T12:30:59.614 INFO:tasks.ceph:[client] debug ms = 1 2024-05-15T12:30:59.614 INFO:tasks.ceph:[client] debug objecter = 20 2024-05-15T12:30:59.615 INFO:tasks.ceph:[client] debug rados = 20 2024-05-15T12:30:59.615 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-15T12:30:59.615 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-05-15T12:30:59.615 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-15T12:30:59.615 INFO:tasks.ceph:[global] ms type = async 2024-05-15T12:30:59.615 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-05-15T12:30:59.615 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-15T12:30:59.615 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-15T12:30:59.615 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-15T12:30:59.615 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-15T12:30:59.615 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-15T12:30:59.615 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-05-15T12:30:59.615 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-05-15T12:30:59.615 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-05-15T12:30:59.615 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-05-15T12:30:59.615 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-05-15T12:30:59.616 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-05-15T12:30:59.616 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-05-15T12:30:59.616 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] debug monc = 20 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-05-15T12:30:59.616 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd class default list = * 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd class load list = * 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-05-15T12:30:59.617 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-05-15T12:30:59.618 INFO:tasks.ceph:Setting up mon.a... 2024-05-15T12:30:59.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-15T12:30:59.679 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.keyring 2024-05-15T12:30:59.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-15T12:30:59.728 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-15T12:30:59.755 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.49'), ('mon.c', '[v2:172.21.15.49:3301,v1:172.21.15.49:6790]'), ('mon.b', '172.21.15.88')] 2024-05-15T12:30:59.755 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.49,[v2:172.21.15.49:3301,v1:172.21.15.49:6790],172.21.15.88', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-05-15T12:30:59.756 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:30:59.756 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-15T12:30:59.811 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.49 --addv c '[v2:172.21.15.49:3301,v1:172.21.15.49:6790]' --add b 172.21.15.88 --print /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:30:59.899 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:30:59.899 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: generated fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:30:59.899 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = quincy 2024-05-15T12:30:59.899 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 0 2024-05-15T12:30:59.899 INFO:teuthology.orchestra.run.smithi049.stdout:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:30:59.899 INFO:teuthology.orchestra.run.smithi049.stdout:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:30:59.899 INFO:teuthology.orchestra.run.smithi049.stdout:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:30:59.900 INFO:teuthology.orchestra.run.smithi049.stdout:min_mon_release 17 (quincy) 2024-05-15T12:30:59.900 INFO:teuthology.orchestra.run.smithi049.stdout:election_strategy: 1 2024-05-15T12:30:59.900 INFO:teuthology.orchestra.run.smithi049.stdout:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:30:59.900 INFO:teuthology.orchestra.run.smithi049.stdout:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:30:59.900 INFO:teuthology.orchestra.run.smithi049.stdout:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:30:59.900 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-15T12:30:59.901 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-15T12:30:59.917 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8874b84e-6b11-44d3-b531-be645feab39a... 2024-05-15T12:30:59.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-15T12:30:59.960 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-15T12:31:00.001 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2024-05-15T12:31:00.001 INFO:teuthology.orchestra.run.smithi049.stdout: chdir = "" 2024-05-15T12:31:00.001 INFO:teuthology.orchestra.run.smithi049.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-15T12:31:00.001 INFO:teuthology.orchestra.run.smithi049.stdout: auth supported = cephx 2024-05-15T12:31:00.001 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: filestore xattr use omap = true 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: mon clock drift allowed = 1.000 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: osd crush chooseleaf type = 0 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: auth debug = true 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on old message = true 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on bug = true 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: mon pg warn max object skew = 0 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.002 INFO:teuthology.orchestra.run.smithi049.stdout: # disable pg_autoscaler by default for new pools 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default size = 2 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow primary affinity = true 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow pg remap = true 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on legacy crush tunables = false 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on crush straw calc version zero = false 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on no sortbitwise = false 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on osd down out interval zero = false 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on too few osds = false 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: mon_allow_pool_size_one = true 2024-05-15T12:31:00.003 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: osd default data pool replay window = 5 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: mon allow pool delete = true 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: mon cluster log file level = debug 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: debug asserts on shutdown = true 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: mon health detail to clog = false 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: mon host = "172.21.15.49,[v2:172.21.15.49:3301,v1:172.21.15.49:6790],172.21.15.88" 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: mon client directed command retry = 5 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: mon election default strategy = 1 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: ms inject socket failures = 5000 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: ms type = async 2024-05-15T12:31:00.004 INFO:teuthology.orchestra.run.smithi049.stdout: osd heartbeat grace = 80 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout:[osd] 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: osd journal size = 100 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub load threshold = 5.0 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub max interval = 600 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock profile = high_recovery_ops 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: osd recover clone overlap = true 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: osd recovery max chunk = 1048576 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug shutdown = true 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug op order = true 2024-05-15T12:31:00.005 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify stray on activate = true 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug trim objects = true 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: osd open classes on start = true 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug pg log writeout = true 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: osd deep scrub update digest min age = 30 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: osd map max advance = 10 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: journal zero on create = true 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: filestore ondisk finisher threads = 3 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: filestore apply finisher threads = 3 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.006 INFO:teuthology.orchestra.run.smithi049.stdout: bdev debug aio = true 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug misdirected ops = true 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: bdev async discard = True 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: bdev enable discard = True 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore allocator = bitmap 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore block size = 96636764160 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore fsck on mount = True 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore zero block detection = True 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluefs = 1 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluestore = 1 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: debug monc = 20 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 20 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: debug rocksdb = 10 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd backfillfull_ratio = 0.85 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd full ratio = 0.9 2024-05-15T12:31:00.007 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd nearfull ratio = 0.8 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd class default list = * 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd class load list = * 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd client watch timeout = 120 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify cached snaps = True 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify missing on start = True 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd failsafe full ratio = 0.95 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd fast shutdown = False 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock override recovery settings = True 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd objectstore = bluestore 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd op queue = debug_random 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd op queue cut off = debug_random 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: osd shutdown pgref assert = True 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr] 2024-05-15T12:31:00.008 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 20 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: mgr/telemetry/nag = false 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout:[mon] 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 20 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: mon data avail warn = 5 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: mon mgr mkfs grace = 240 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2024-05-15T12:31:00.009 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd reporter subtree level = osd 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd prime pg temp = true 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: auth mon ticket ttl = 660 # 11m 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: auth service ticket ttl = 240 # 4m 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: # don't complain about insecure global_id in the test suite 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: # 1m isn't quite enough 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: mon_down_mkfs_grace = 2m 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.010 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_filestore_osds = false 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: mon min osdmap epochs = 50 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd crush smoke test = False 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: mon osdmap full prune interval = 2 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: mon osdmap full prune min = 15 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: mon osdmap full prune txsize = 2 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: mon scrub interval = 300 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on pool no app = False 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: paxos service trim min = 10 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout:[client] 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: rgw cache enabled = true 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable ops log = true 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable usage log = true 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-15T12:31:00.011 INFO:teuthology.orchestra.run.smithi049.stdout: debug monc = 20 2024-05-15T12:31:00.012 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2024-05-15T12:31:00.012 INFO:teuthology.orchestra.run.smithi049.stdout: debug objecter = 20 2024-05-15T12:31:00.012 INFO:teuthology.orchestra.run.smithi049.stdout: debug rados = 20 2024-05-15T12:31:00.012 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.a] 2024-05-15T12:31:00.012 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.c] 2024-05-15T12:31:00.012 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.b] 2024-05-15T12:31:00.012 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: chdir = "" 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: auth supported = cephx 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: filestore xattr use omap = true 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: mon clock drift allowed = 1.000 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: osd crush chooseleaf type = 0 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: auth debug = true 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on old message = true 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on bug = true 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.013 INFO:teuthology.orchestra.run.smithi088.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: mon pg warn max object skew = 0 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: # disable pg_autoscaler by default for new pools 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default size = 2 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow primary affinity = true 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow pg remap = true 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on legacy crush tunables = false 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on crush straw calc version zero = false 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on no sortbitwise = false 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on osd down out interval zero = false 2024-05-15T12:31:00.014 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on too few osds = false 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: mon_allow_pool_size_one = true 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: osd default data pool replay window = 5 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: mon allow pool delete = true 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: mon cluster log file level = debug 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: debug asserts on shutdown = true 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: mon health detail to clog = false 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: mon host = "172.21.15.49,[v2:172.21.15.49:3301,v1:172.21.15.49:6790],172.21.15.88" 2024-05-15T12:31:00.015 INFO:teuthology.orchestra.run.smithi088.stdout: mon client directed command retry = 5 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: mon election default strategy = 1 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject socket failures = 5000 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: ms type = async 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: osd heartbeat grace = 80 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout:[osd] 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: osd journal size = 100 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub load threshold = 5.0 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub max interval = 600 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock profile = high_recovery_ops 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: osd recover clone overlap = true 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: osd recovery max chunk = 1048576 2024-05-15T12:31:00.016 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug shutdown = true 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug op order = true 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify stray on activate = true 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug trim objects = true 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: osd open classes on start = true 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug pg log writeout = true 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: osd deep scrub update digest min age = 30 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: osd map max advance = 10 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: journal zero on create = true 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.017 INFO:teuthology.orchestra.run.smithi088.stdout: filestore ondisk finisher threads = 3 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: filestore apply finisher threads = 3 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: bdev debug aio = true 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug misdirected ops = true 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: bdev async discard = True 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: bdev enable discard = True 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore allocator = bitmap 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore block size = 96636764160 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore fsck on mount = True 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore zero block detection = True 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluefs = 1 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluestore = 1 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: debug monc = 20 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-05-15T12:31:00.018 INFO:teuthology.orchestra.run.smithi088.stdout: debug osd = 20 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: debug rocksdb = 10 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd backfillfull_ratio = 0.85 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd full ratio = 0.9 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd nearfull ratio = 0.8 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd class default list = * 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd class load list = * 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd client watch timeout = 120 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify cached snaps = True 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify missing on start = True 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd failsafe full ratio = 0.95 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd fast shutdown = False 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock override recovery settings = True 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd objectstore = bluestore 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd op queue = debug_random 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd op queue cut off = debug_random 2024-05-15T12:31:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: osd shutdown pgref assert = True 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr] 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: debug mgr = 20 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: mgr/telemetry/nag = false 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout:[mon] 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: debug paxos = 20 2024-05-15T12:31:00.020 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: mon data avail warn = 5 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: mon mgr mkfs grace = 240 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd reporter subtree level = osd 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd prime pg temp = true 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: auth mon ticket ttl = 660 # 11m 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: auth service ticket ttl = 240 # 4m 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: # don't complain about insecure global_id in the test suite 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.021 INFO:teuthology.orchestra.run.smithi088.stdout: # 1m isn't quite enough 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: mon_down_mkfs_grace = 2m 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_filestore_osds = false 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: mon min osdmap epochs = 50 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd crush smoke test = False 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune interval = 2 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune min = 15 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune txsize = 2 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: mon scrub interval = 300 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on pool no app = False 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: paxos service trim min = 10 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout:[client] 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: rgw cache enabled = true 2024-05-15T12:31:00.022 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable ops log = true 2024-05-15T12:31:00.023 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable usage log = true 2024-05-15T12:31:00.023 INFO:teuthology.orchestra.run.smithi088.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-15T12:31:00.023 INFO:teuthology.orchestra.run.smithi088.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-15T12:31:00.023 INFO:teuthology.orchestra.run.smithi088.stdout: debug monc = 20 2024-05-15T12:31:00.023 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-05-15T12:31:00.023 INFO:teuthology.orchestra.run.smithi088.stdout: debug objecter = 20 2024-05-15T12:31:00.023 INFO:teuthology.orchestra.run.smithi088.stdout: debug rados = 20 2024-05-15T12:31:00.023 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.a] 2024-05-15T12:31:00.023 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.c] 2024-05-15T12:31:00.023 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.b] 2024-05-15T12:31:00.024 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-15T12:31:00.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-15T12:31:00.120 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-15T12:31:00.120 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:00.120 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-15T12:31:00.136 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:00.136 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-15T12:31:00.192 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T12:31:00.192 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:00.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-15T12:31:00.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-15T12:31:00.267 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:00.267 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:00.323 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T12:31:00.323 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:00.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-15T12:31:00.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-15T12:31:00.358 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:00.358 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:00.413 INFO:tasks.ceph:Setting up mon nodes... 2024-05-15T12:31:00.413 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-15T12:31:00.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-15T12:31:00.465 INFO:teuthology.orchestra.run.smithi049.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-15T12:31:00.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-15T12:31:00.544 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-15T12:31:00.547 INFO:tasks.ceph:Setting up mds nodes... 2024-05-15T12:31:00.547 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-15T12:31:00.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-15T12:31:00.590 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-15T12:31:00.602 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-15T12:31:00.643 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-15T12:31:00.655 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-15T12:31:00.655 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-15T12:31:00.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-15T12:31:00.681 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-15T12:31:00.681 INFO:tasks.ceph:role: osd.0 2024-05-15T12:31:00.681 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T12:31:00.681 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-15T12:31:00.746 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:31:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:31:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:31:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:31:00.746 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:31:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:31:00.746 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:31:00.746 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:31:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:31:00.747 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:31:00.778 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-05-15T12:31:00.780 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-05-15T12:31:00.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-15T12:31:00.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-15T12:31:00.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-15T12:31:00.986 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-15T12:31:00.986 INFO:tasks.ceph:role: osd.1 2024-05-15T12:31:00.986 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T12:31:00.986 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-15T12:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:31:01.051 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:31:01.083 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-05-15T12:31:01.085 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-05-15T12:31:01.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-15T12:31:01.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-15T12:31:01.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-15T12:31:01.296 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-15T12:31:01.296 INFO:tasks.ceph:role: osd.2 2024-05-15T12:31:01.296 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T12:31:01.296 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-15T12:31:01.361 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:31:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:31:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:31:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:31:01.362 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:31:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:31:01.362 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:31:01.362 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:31:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:31:01.363 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:31:01.395 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-05-15T12:31:01.397 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-05-15T12:31:01.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-15T12:31:01.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-15T12:31:01.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-15T12:31:01.601 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-15T12:31:01.602 INFO:tasks.ceph:role: osd.3 2024-05-15T12:31:01.602 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T12:31:01.602 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-15T12:31:01.667 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:31:01.667 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:31:01.667 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:31:01.667 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:31:01.667 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:31:01.667 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:31:01.667 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:31:01.668 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:31:01.668 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:31:01.668 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:31:01.700 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-05-15T12:31:01.703 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-05-15T12:31:01.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-15T12:31:01.773 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-15T12:31:01.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:01.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:01.953+0000 7fdb54421640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-15T12:31:01.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:01.953+0000 7fdb54421640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-15T12:31:01.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:01.953+0000 7fdb54421640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-15T12:31:01.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:01.953+0000 7fdb54421640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-15T12:31:01.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:01.953+0000 7fdb54421640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-15T12:31:01.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:01.954+0000 7fdb54421640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-15T12:31:04.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-15T12:31:04.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:04.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:04.923+0000 7f75abc01640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-15T12:31:04.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:04.923+0000 7f75abc01640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-15T12:31:04.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:04.923+0000 7f75abc01640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-15T12:31:04.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:04.923+0000 7f75abc01640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-15T12:31:04.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:04.923+0000 7f75abc01640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-15T12:31:04.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:04.924+0000 7f75abc01640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-15T12:31:07.759 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-15T12:31:07.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:07.892+0000 7fcc3dd1c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-15T12:31:07.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:07.892+0000 7fcc3dd1c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-15T12:31:07.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:07.892+0000 7fcc3dd1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-15T12:31:07.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:07.892+0000 7fcc3dd1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-15T12:31:07.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:07.892+0000 7fcc3dd1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-15T12:31:07.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:07.893+0000 7fcc3dd1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-15T12:31:10.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-15T12:31:10.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:10.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:10.862+0000 7f34fd2a8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-15T12:31:10.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:10.862+0000 7f34fd2a8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-15T12:31:10.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:10.862+0000 7f34fd2a8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-15T12:31:10.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:10.862+0000 7f34fd2a8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-15T12:31:10.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:10.862+0000 7f34fd2a8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-15T12:31:10.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:10.863+0000 7f34fd2a8640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-15T12:31:13.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-15T12:31:13.735 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-15T12:31:13.764 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-15T12:31:13.764 INFO:tasks.ceph:role: osd.4 2024-05-15T12:31:13.764 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T12:31:13.765 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-15T12:31:13.840 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:31:13.840 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:31:13.840 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:31:13.840 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:31:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:31:13.841 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:31:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:31:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:31:13.841 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:31:13.841 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:31:14.341 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-05-15T12:31:14.344 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-05-15T12:31:14.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-15T12:31:14.478 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-15T12:31:14.511 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-15T12:31:14.578 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-15T12:31:14.578 INFO:tasks.ceph:role: osd.5 2024-05-15T12:31:14.578 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T12:31:14.579 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-15T12:31:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:31:14.652 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:31:14.652 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:31:14.652 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:31:14.652 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:31:14.652 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:31:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:31:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:31:14.653 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:31:14.653 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:31:15.098 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-05-15T12:31:15.100 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-05-15T12:31:15.101 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-15T12:31:15.190 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-15T12:31:15.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-15T12:31:15.287 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-15T12:31:15.287 INFO:tasks.ceph:role: osd.6 2024-05-15T12:31:15.287 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T12:31:15.287 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-15T12:31:15.359 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:31:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:31:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:31:15.359 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:31:15.360 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:31:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:31:15.360 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:31:15.360 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:31:15.360 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:31:15.360 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:31:15.865 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-05-15T12:31:15.867 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-05-15T12:31:15.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-15T12:31:15.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-15T12:31:16.018 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-15T12:31:16.085 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-15T12:31:16.085 INFO:tasks.ceph:role: osd.7 2024-05-15T12:31:16.085 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T12:31:16.085 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-15T12:31:16.160 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-15T12:31:16.160 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-15T12:31:16.160 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-15T12:31:16.160 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-15T12:31:16.160 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-15T12:31:16.160 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-05-15T12:31:16.160 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-15T12:31:16.160 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-15T12:31:16.160 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-15T12:31:16.160 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-15T12:31:16.676 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-05-15T12:31:16.678 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-05-15T12:31:16.678 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-15T12:31:16.814 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-15T12:31:16.847 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:16.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:16.950+0000 7fbda442e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-15T12:31:16.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:16.950+0000 7fbda442e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-15T12:31:16.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:16.950+0000 7fbda442e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-15T12:31:16.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:16.950+0000 7fbda442e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-15T12:31:16.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:16.950+0000 7fbda442e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-15T12:31:16.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:16.951+0000 7fbda442e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-15T12:31:19.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-15T12:31:19.820 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:19.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:19.919+0000 7f0f8d21a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-15T12:31:19.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:19.919+0000 7f0f8d21a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-15T12:31:19.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:19.919+0000 7f0f8d21a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-15T12:31:19.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:19.919+0000 7f0f8d21a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-15T12:31:19.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:19.919+0000 7f0f8d21a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-15T12:31:19.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:19.920+0000 7f0f8d21a640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-15T12:31:22.755 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-15T12:31:22.785 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:22.884+0000 7f4e61415640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-15T12:31:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:22.884+0000 7f4e61415640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-15T12:31:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:22.884+0000 7f4e61415640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-15T12:31:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:22.884+0000 7f4e61415640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-15T12:31:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:22.884+0000 7f4e61415640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-15T12:31:22.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:22.885+0000 7f4e61415640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-15T12:31:25.729 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-15T12:31:25.759 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:25.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:25.857+0000 7f45b8613640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-15T12:31:25.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:25.858+0000 7f45b8613640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-15T12:31:25.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:25.858+0000 7f45b8613640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-15T12:31:25.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:25.858+0000 7f45b8613640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-15T12:31:25.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:25.858+0000 7f45b8613640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-15T12:31:25.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:25.858+0000 7f45b8613640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-15T12:31:28.687 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-15T12:31:28.716 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-15T12:31:28.716 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:28.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-15T12:31:28.748 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:28.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-15T12:31:28.818 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:28.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-15T12:31:28.887 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:28.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-15T12:31:28.955 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:28.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-15T12:31:29.023 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:29.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-15T12:31:29.053 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:29.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-15T12:31:29.122 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:29.122 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-15T12:31:29.190 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:29.190 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-15T12:31:29.259 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:29.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-15T12:31:29.325 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:29.325 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-15T12:31:29.344 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:29.344 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-15T12:31:29.383 INFO:tasks.ceph:Adding keys to all mons... 2024-05-15T12:31:29.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-15T12:31:29.387 DEBUG:teuthology.orchestra.run.smithi088:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-15T12:31:29.427 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr.y] 2024-05-15T12:31:29.427 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAEq0RmHOpyHBAAIrIgvfLNrnZL9S2pllkFxA== 2024-05-15T12:31:29.427 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2024-05-15T12:31:29.427 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAFq0Rm2EbvOBAAURveecutDlPM0GdcsBTxxQ== 2024-05-15T12:31:29.427 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2024-05-15T12:31:29.427 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAIq0RmVYkkNxAAmACQLyesbDt6MCJSlXFWYw== 2024-05-15T12:31:29.427 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2024-05-15T12:31:29.427 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQALq0RmFXhINRAAlQ8F4Gm/vT64XVvndlqDvw== 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.3] 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAOq0RmxwWCMxAAzaZO8FxOJpIOaidHrCXHCg== 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr.x] 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAEq0RmAXojIRAA2NcgKOpqG5WXFdd8ET73tA== 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.4] 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAUq0Rmgme7OBAAgfDpBwqeEMvg+V9a8kzu5Q== 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.5] 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAXq0RmSC3kNhAAkwa3hzvucxPLwKS6URGlAg== 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.6] 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAaq0RmEl7MNBAAc04MBprl3Qdstn4SHJTAsA== 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.7] 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAdq0RmSRI2MxAAijyVi21HZGrzm8VljWUu5w== 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAEq0RmGLPaIxAADqgMF2rmzj4zQPN378vjnA== 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout:[client.1] 2024-05-15T12:31:29.428 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAEq0RmjTMFJxAAKcdF8cBNUYbtTL9EryJQQg== 2024-05-15T12:31:29.448 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.y] 2024-05-15T12:31:29.448 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAEq0RmHOpyHBAAIrIgvfLNrnZL9S2pllkFxA== 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.0] 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAFq0Rm2EbvOBAAURveecutDlPM0GdcsBTxxQ== 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.1] 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAIq0RmVYkkNxAAmACQLyesbDt6MCJSlXFWYw== 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.2] 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQALq0RmFXhINRAAlQ8F4Gm/vT64XVvndlqDvw== 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.3] 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAOq0RmxwWCMxAAzaZO8FxOJpIOaidHrCXHCg== 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.x] 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAEq0RmAXojIRAA2NcgKOpqG5WXFdd8ET73tA== 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.4] 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAUq0Rmgme7OBAAgfDpBwqeEMvg+V9a8kzu5Q== 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.5] 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAXq0RmSC3kNhAAkwa3hzvucxPLwKS6URGlAg== 2024-05-15T12:31:29.449 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.6] 2024-05-15T12:31:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAaq0RmEl7MNBAAc04MBprl3Qdstn4SHJTAsA== 2024-05-15T12:31:29.450 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.7] 2024-05-15T12:31:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAdq0RmSRI2MxAAijyVi21HZGrzm8VljWUu5w== 2024-05-15T12:31:29.450 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-05-15T12:31:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAEq0RmGLPaIxAADqgMF2rmzj4zQPN378vjnA== 2024-05-15T12:31:29.450 INFO:teuthology.orchestra.run.smithi088.stdout:[client.1] 2024-05-15T12:31:29.450 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAEq0RmjTMFJxAAKcdF8cBNUYbtTL9EryJQQg== 2024-05-15T12:31:29.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-15T12:31:29.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-15T12:31:29.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.547 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.656 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.709 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.759 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-15T12:31:29.761 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-15T12:31:29.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.814 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.866 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:29.971 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-15T12:31:30.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-15T12:31:30.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-15T12:31:30.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-15T12:31:30.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-15T12:31:30.132 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-15T12:31:30.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-15T12:31:30.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-15T12:31:30.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-15T12:31:30.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-15T12:31:30.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-15T12:31:30.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-15T12:31:30.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-15T12:31:30.494 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-15T12:31:30.592 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-15T12:31:30.616 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:30.619 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-15T12:31:30.671 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-15T12:31:30.672 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-05-15T12:31:30.672 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-15T12:31:30.672 DEBUG:teuthology.orchestra.run.smithi049:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-05-15T12:31:30.674 INFO:tasks.ceph.mon.a:Started 2024-05-15T12:31:30.674 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-05-15T12:31:30.674 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-15T12:31:30.674 DEBUG:teuthology.orchestra.run.smithi049:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-05-15T12:31:30.676 INFO:tasks.ceph.mon.c:Started 2024-05-15T12:31:30.676 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-05-15T12:31:30.677 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-15T12:31:30.677 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-05-15T12:31:30.714 INFO:tasks.ceph.mon.b:Started 2024-05-15T12:31:30.714 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-15T12:31:30.715 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-15T12:31:30.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-05-15T12:31:30.717 INFO:tasks.ceph.mgr.y:Started 2024-05-15T12:31:30.717 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-15T12:31:30.718 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-05-15T12:31:30.719 INFO:tasks.ceph.mgr.x:Started 2024-05-15T12:31:30.720 INFO:tasks.ceph:Setting crush tunables to default 2024-05-15T12:31:30.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd crush tunables default 2024-05-15T12:31:30.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.834+0000 7f426584b640 10 monclient: get_monmap_and_config 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.834+0000 7f426584b640 10 monclient: build_initial_monmap 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.834+0000 7f426584b640 10 monclient: monmap: 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:31:30.835202+0000 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:31:30.835202+0000 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:30.824 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.835+0000 7f426584b640 1 Processor -- start 2024-05-15T12:31:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.835+0000 7f426584b640 1 -- start start 2024-05-15T12:31:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.835+0000 7f426584b640 10 monclient: init 2024-05-15T12:31:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.835+0000 7f426584b640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.835+0000 7f426584b640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-05-15T12:31:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42600b5410 addr v1:172.21.15.88:6789/0 2024-05-15T12:31:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 0x7f42600a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 10 monclient(hunting): picked mon.noname-b con 0x7f42601400a0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f4260146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 10 monclient(hunting): picked mon.noname-a con 0x7f4260145c00 addr v2:172.21.15.49:3300/0 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42600ab470 con 0x7f42600b5410 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42600b57f0 con 0x7f4260145c00 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42600a9ea0 con 0x7f42601400a0 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f426584b640 10 monclient(hunting): authenticate will time out at 1448.997723s 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f425effd640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f42600b5410 legacy=0x7f426013f990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f425f7fe640 1 -- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 msgr2=0x7f4260146070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3300/0 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f425e7fc640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 msgr2=0x7f42600a9960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3301/0 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f425f7fe640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f4260146070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:31:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:30.836+0000 7f425e7fc640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 0x7f42600a9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:31:31.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:31.037+0000 7f425e7fc640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 msgr2=0x7f42600a9960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3301/0 2024-05-15T12:31:31.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:31.037+0000 7f425f7fe640 1 -- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 msgr2=0x7f4260146070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3300/0 2024-05-15T12:31:31.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:31.037+0000 7f425e7fc640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 0x7f42600a9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:31:31.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:31.037+0000 7f425f7fe640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f4260146070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:31:31.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:31.037+0000 7f425effd640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f42600b5410 legacy=0x7f426013f990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-05-15T12:31:31.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:31.438+0000 7f425e7fc640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 msgr2=0x7f42600a9960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3301/0 2024-05-15T12:31:31.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:31.438+0000 7f425f7fe640 1 -- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 msgr2=0x7f4260146070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3300/0 2024-05-15T12:31:31.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:31.438+0000 7f425effd640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f42600b5410 legacy=0x7f426013f990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-05-15T12:31:31.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:31.438+0000 7f425e7fc640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 0x7f42600a9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:31:31.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:31.438+0000 7f425f7fe640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f4260146070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:31:32.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:32.239+0000 7f425e7fc640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 msgr2=0x7f42600a9960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3301/0 2024-05-15T12:31:32.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:32.239+0000 7f425f7fe640 1 -- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 msgr2=0x7f4260146070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3300/0 2024-05-15T12:31:32.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:32.239+0000 7f425e7fc640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 0x7f42600a9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:31:32.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:32.239+0000 7f425f7fe640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f4260146070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:31:32.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:32.239+0000 7f425effd640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f42600b5410 legacy=0x7f426013f990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-05-15T12:31:33.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 10 monclient(hunting): tick 2024-05-15T12:31:33.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 10 monclient(hunting): _check_auth_tickets 2024-05-15T12:31:33.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 1 monclient(hunting): continuing hunt 2024-05-15T12:31:33.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 10 monclient(hunting): _reopen_session rank -1 2024-05-15T12:31:33.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 msgr2=0x7f42600a9960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:31:33.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42601400a0 0x7f42600a9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:33.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 1 -- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 msgr2=0x7f4260146070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:31:33.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f4260146070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:33.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7f42600b5410 legacy=0x7f426013f990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 10 monclient: _add_conns ranks=[0,4] 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f424c0036f0 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 0x7f424c009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f424c0073a0 addr v2:172.21.15.88:3300/0 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42600aa0c0 con 0x7f424c0036f0 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42600b57f0 con 0x7f424c0073a0 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425cff9640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425effd640 1 -- >> v1:172.21.15.49:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.49:6789/0 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425e7fc640 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2024-05-15T12:31:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:33.836+0000 7f425e7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:31:34.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:34.037+0000 7f425effd640 1 -- >> v1:172.21.15.49:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.49:6789/0 2024-05-15T12:31:34.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:34.037+0000 7f425e7fc640 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2024-05-15T12:31:34.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:34.037+0000 7f425e7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:31:34.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:34.438+0000 7f425effd640 1 -- >> v1:172.21.15.49:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.49:6789/0 2024-05-15T12:31:34.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:34.438+0000 7f425e7fc640 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2024-05-15T12:31:34.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:34.438+0000 7f425e7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:31:35.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:35.239+0000 7f425effd640 1 -- >> v1:172.21.15.49:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.49:6789/0 2024-05-15T12:31:35.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:35.239+0000 7f425e7fc640 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:3300/0 2024-05-15T12:31:35.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:35.239+0000 7f425e7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:31:36.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 10 monclient(hunting): tick 2024-05-15T12:31:36.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 10 monclient(hunting): _check_auth_tickets 2024-05-15T12:31:36.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 1 monclient(hunting): continuing hunt 2024-05-15T12:31:36.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 10 monclient(hunting): _reopen_session rank -1 2024-05-15T12:31:36.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 1 -- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:31:36.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f424c0073a0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:36.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 1 -- >> v1:172.21.15.49:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425f7fe640 1 -- reap_dead start 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f424c009f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f4260145c00 addr v2:172.21.15.49:3300/0 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f424c0036f0 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 0x7f424c009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f424c00a4c0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42600ab470 con 0x7f424c0036f0 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42600b57f0 con 0x7f4260145c00 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42600a9ea0 con 0x7f424c00a4c0 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425cff9640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:36.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425effd640 1 -- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 msgr2=0x7f424c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3300/0 2024-05-15T12:31:36.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425e7fc640 1 -- >> v1:172.21.15.49:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.49:6789/0 2024-05-15T12:31:36.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425f7fe640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3301/0 2024-05-15T12:31:36.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425effd640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f424c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:31:36.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:36.835+0000 7f425f7fe640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:31:37.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:37.036+0000 7f425e7fc640 1 -- >> v1:172.21.15.49:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.49:6789/0 2024-05-15T12:31:37.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:37.036+0000 7f425f7fe640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3301/0 2024-05-15T12:31:37.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:37.036+0000 7f425effd640 1 -- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 msgr2=0x7f424c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3300/0 2024-05-15T12:31:37.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:37.036+0000 7f425effd640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f424c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:31:37.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:37.036+0000 7f425f7fe640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:31:37.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:37.437+0000 7f425e7fc640 1 -- >> v1:172.21.15.49:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.49:6789/0 2024-05-15T12:31:37.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:37.437+0000 7f425effd640 1 -- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 msgr2=0x7f424c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3300/0 2024-05-15T12:31:37.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:37.437+0000 7f425f7fe640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3301/0 2024-05-15T12:31:37.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:37.437+0000 7f425effd640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f424c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:31:37.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:37.437+0000 7f425f7fe640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:31:37.771 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:00:06.939 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:31:38.079 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:00:07.246 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:31:38.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:38.238+0000 7f425effd640 1 -- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 msgr2=0x7f424c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3300/0 2024-05-15T12:31:38.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:38.238+0000 7f425e7fc640 1 -- >> v1:172.21.15.49:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.49:6789/0 2024-05-15T12:31:38.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:38.238+0000 7f425f7fe640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.49:3301/0 2024-05-15T12:31:38.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:38.238+0000 7f425effd640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f424c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:31:38.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:38.238+0000 7f425f7fe640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:31:38.282 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:00:07.491 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:31:38.282 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:00:07.491 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:31:38.608 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:00:07.817 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:31:38.609 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:00:07.819 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:31:39.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 10 monclient(hunting): tick 2024-05-15T12:31:39.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 10 monclient(hunting): _check_auth_tickets 2024-05-15T12:31:39.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 1 monclient(hunting): continuing hunt 2024-05-15T12:31:39.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 10 monclient(hunting): _reopen_session rank -1 2024-05-15T12:31:39.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:31:39.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00a4c0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:39.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 1 -- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 msgr2=0x7f424c009f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:31:39.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4260145c00 0x7f424c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:39.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 1 -- >> v1:172.21.15.49:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:31:39.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 10 monclient: _add_conns ranks=[4,1] 2024-05-15T12:31:39.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:39.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f424c00d0c0 addr v2:172.21.15.88:3300/0 2024-05-15T12:31:39.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f424c00f9f0 addr v1:172.21.15.88:6789/0 2024-05-15T12:31:39.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42600aa0c0 con 0x7f424c00f9f0 2024-05-15T12:31:39.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:39.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42600b57f0 con 0x7f424c00d0c0 2024-05-15T12:31:39.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.835+0000 7f425cff9640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:39.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.899+0000 7f425e7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f424c00f9f0 0x7f424c011de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:57272/0 (socket says 172.21.15.49:57272) 2024-05-15T12:31:39.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.899+0000 7f425e7fc640 1 -- 172.21.15.49:0/2131274978 learned_addr learned my addr 172.21.15.49:0/2131274978 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:31:39.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.942+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:39.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.943+0000 7f425effd640 10 monclient(hunting): get_auth_request con 0x7f424c00d0c0 auth_method 0 2024-05-15T12:31:39.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.943+0000 7f425effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:39.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.943+0000 7f425effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:39.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.943+0000 7f425effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:39.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.947+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 msgr2=0x7f424c00f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-15T12:31:39.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.947+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 msgr2=0x7f424c00f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:31:39.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.947+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:31:39.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:39.947+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:31:40.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.148+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:40.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.149+0000 7f425effd640 10 monclient(hunting): get_auth_request con 0x7f424c00d0c0 auth_method 0 2024-05-15T12:31:40.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.149+0000 7f425effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:40.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.149+0000 7f425effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:40.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.149+0000 7f425effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:40.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.149+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 msgr2=0x7f424c00f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-15T12:31:40.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.149+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 msgr2=0x7f424c00f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:31:40.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.149+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:31:40.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.149+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:31:40.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.722+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:40.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.723+0000 7f425effd640 10 monclient(hunting): get_auth_request con 0x7f424c00d0c0 auth_method 0 2024-05-15T12:31:40.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.723+0000 7f425effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:40.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.723+0000 7f425effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:40.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.723+0000 7f425effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:40.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.734+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 msgr2=0x7f424c00f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-15T12:31:40.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.734+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 msgr2=0x7f424c00f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:31:40.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.734+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:31:40.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:40.734+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-15T12:31:41.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:41.535+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:41.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:41.535+0000 7f425effd640 10 monclient(hunting): get_auth_request con 0x7f424c00d0c0 auth_method 0 2024-05-15T12:31:41.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:41.535+0000 7f425effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:41.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:41.535+0000 7f425effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:41.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:41.535+0000 7f425effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:41.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:41.536+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 msgr2=0x7f424c00f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-15T12:31:41.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:41.536+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 msgr2=0x7f424c00f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:31:41.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:41.536+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:31:41.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:41.536+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 10 monclient(hunting): tick 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 10 monclient(hunting): _check_auth_tickets 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 1 monclient(hunting): continuing hunt 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 10 monclient(hunting): _reopen_session rank -1 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 1 -- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 msgr2=0x7f424c00f4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 1 --2- 172.21.15.49:0/2131274978 >> v2:172.21.15.88:3300/0 conn(0x7f424c00d0c0 0x7f424c00f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 1 -- 172.21.15.49:0/2131274978 >> v1:172.21.15.88:6789/0 conn(0x7f424c00f9f0 legacy=0x7f424c011de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425f7fe640 1 -- 172.21.15.49:0/2131274978 reap_dead start 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f424c0036f0 addr v1:172.21.15.88:6789/0 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f424c009f90 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424c009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f424c00f4c0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 1 -- 172.21.15.49:0/2131274978 --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42600ab470 con 0x7f424c009f90 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 1 -- 172.21.15.49:0/2131274978 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424c0070c0 con 0x7f424c0036f0 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 1 -- 172.21.15.49:0/2131274978 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42600b57f0 con 0x7f424c00f4c0 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.835+0000 7f425cff9640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.836+0000 7f425f7fe640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424c009860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.836+0000 7f425f7fe640 10 monclient(hunting): get_auth_request con 0x7f424c00f4c0 auth_method 0 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.836+0000 7f425f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.836+0000 7f425f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:42.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.836+0000 7f425f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:42.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.837+0000 7f425f7fe640 1 -- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-15T12:31:42.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.837+0000 7f425f7fe640 1 -- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:31:42.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.837+0000 7f425f7fe640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424c009860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:31:42.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:42.837+0000 7f425f7fe640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:31:43.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.072+0000 7f425f7fe640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424c009860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:43.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.072+0000 7f425f7fe640 10 monclient(hunting): get_auth_request con 0x7f424c00f4c0 auth_method 0 2024-05-15T12:31:43.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.072+0000 7f425f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:43.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.072+0000 7f425f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:43.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.072+0000 7f425f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:43.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.106+0000 7f425f7fe640 1 -- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-15T12:31:43.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.106+0000 7f425f7fe640 1 -- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T12:31:43.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.106+0000 7f425f7fe640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424c009860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:31:43.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.106+0000 7f425f7fe640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-15T12:31:43.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.357+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211154500 0 0) 0x7f42600ab470 con 0x7f424c009f90 2024-05-15T12:31:43.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.357+0000 7f425dffb640 10 monclient(hunting): _negotiate 2024-05-15T12:31:43.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.357+0000 7f425dffb640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:43.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.357+0000 7f425dffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:43.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.357+0000 7f425dffb640 10 monclient(hunting): my global_id is 4098 2024-05-15T12:31:43.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.357+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4248003520 con 0x7f424c009f90 2024-05-15T12:31:43.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.406+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875994418 0 0) 0x7f424c0070c0 con 0x7f424c0036f0 2024-05-15T12:31:43.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.406+0000 7f425dffb640 10 monclient(hunting): _negotiate 2024-05-15T12:31:43.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.406+0000 7f425dffb640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:43.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.406+0000 7f425dffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:43.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.406+0000 7f425dffb640 10 monclient(hunting): my global_id is 4099 2024-05-15T12:31:43.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.406+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42600ab470 con 0x7f424c0036f0 2024-05-15T12:31:43.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 <== mon.0 v1:172.21.15.49:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 597324988 0 0) 0x7f4248003520 con 0x7f424c009f90 2024-05-15T12:31:43.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 msgr2=0x7f424c009860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T12:31:43.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424c009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:43.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 >> v1:172.21.15.88:6789/0 conn(0x7f424c0036f0 legacy=0x7f424c0062a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:43.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient: _finish_hunting 0 2024-05-15T12:31:43.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 monclient: found mon.noname-a-legacy 2024-05-15T12:31:43.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:43.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.49:6789/0 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 --> v1:172.21.15.49:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c005ae0 con 0x7f424c009f90 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient: _finish_auth 0 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient: _check_auth_tickets 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 <== mon.0 v1:172.21.15.49:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2686122088 0 0) 0x7f4254004240 con 0x7f424c009f90 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f426584b640 5 monclient: authenticate success, global_id 4098 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f426584b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient: dump: 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:43.442 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] but i'm connected to v1:172.21.15.49:6789/0, reconnecting 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 >> v1:172.21.15.49:6789/0 conn(0x7f424c009f90 legacy=0x7f424c0069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4248003be0 0x7f4248003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient(hunting): picked mon.b con 0x7f4248003be0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4248004800 0x7f4248004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:43.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient(hunting): picked mon.a con 0x7f4248004800 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient(hunting): picked mon.c con 0x7f424c00f4c0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42600b57f0 con 0x7f4248004800 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4248005520 con 0x7f4248003be0 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424800bf00 con 0x7f424c00f4c0 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425dffb640 10 monclient(hunting): _finish_auth 0 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f426584b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.450+0000 7f425f7fe640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.451+0000 7f425f7fe640 10 monclient(hunting): get_auth_request con 0x7f424c00f4c0 auth_method 0 2024-05-15T12:31:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.451+0000 7f425f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.451+0000 7f425f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.451+0000 7f425f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.452+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4248003be0 0x7f4248003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.452+0000 7f425effd640 10 monclient(hunting): get_auth_request con 0x7f4248003be0 auth_method 0 2024-05-15T12:31:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.452+0000 7f425effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.452+0000 7f425effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.452+0000 7f425effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:43.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.454+0000 7f425effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:43.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.454+0000 7f425effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:43.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.454+0000 7f425effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:31:43.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.457+0000 7f425f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:43.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.457+0000 7f425f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:43.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.457+0000 7f425f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:31:43.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-05-15T12:31:43.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 msgr2=0x7f424800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f424c00f4c0 0x7f424800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4248004800 msgr2=0x7f4248004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4248004800 0x7f4248004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 10 monclient: _finish_hunting 0 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 1 monclient: found mon.b 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 1 -- 172.21.15.49:0/2131274978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4248003520 con 0x7f4248003be0 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 10 monclient: _finish_auth 0 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 10 monclient: _check_auth_tickets 2024-05-15T12:31:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.499+0000 7f425effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:43.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.502+0000 7f425effd640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4248003be0 0x7f4248003fe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4250009400 tx=0x7f4248005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7350125d4e8ce60e server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:43.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250009b80 con 0x7f4248003be0 2024-05-15T12:31:43.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:43.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:43.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425dffb640 10 monclient: dump: 2024-05-15T12:31:43.532 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:31:43.532 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:43.532 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:43.532 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:43.532 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:43.532 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425002e670 con 0x7f4248003be0 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425dffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425dffb640 1 -- 172.21.15.49:0/2131274978 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425002eaa0 con 0x7f4248003be0 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 4 set_mon_vals no callback set 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425dffb640 10 monclient: dump: 2024-05-15T12:31:43.533 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:31:43.534 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:43.534 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:43.534 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:43.534 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:43.534 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:43.534 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:43.534 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:43.534 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:43.534 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 10 monclient: shutdown 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 1 -- 172.21.15.49:0/2131274978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4248003be0 msgr2=0x7f4248003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 1 --2- 172.21.15.49:0/2131274978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4248003be0 0x7f4248003fe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4250009400 tx=0x7f4248005830 comp rx=0 tx=0).stop 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f425f7fe640 1 -- 172.21.15.49:0/2131274978 reap_dead start 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 1 -- 172.21.15.49:0/2131274978 shutdown_connections 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 1 -- 172.21.15.49:0/2131274978 >> 172.21.15.49:0/2131274978 conn(0x7f4260066f00 msgr2=0x7f4260067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 1 -- 172.21.15.49:0/2131274978 shutdown_connections 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.542+0000 7f426584b640 1 -- 172.21.15.49:0/2131274978 wait complete. 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.543+0000 7f426584b640 10 monclient: build_initial_monmap 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.543+0000 7f426584b640 10 monclient: monmap: 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:43.537 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 1 librados: starting msgr at 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 1 librados: starting objecter 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 1 Processor -- start 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 1 -- start start 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 1 librados: setting wanted keys 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 1 librados: calling monclient init 2024-05-15T12:31:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 10 monclient: init 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4260145410 0x7f42601d88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 10 monclient(hunting): picked mon.noname-a con 0x7f4260145410 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42601d7f20 0x7f42601db420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 10 monclient(hunting): picked mon.noname-b con 0x7f42601d7f20 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4260143300 0x7f42600b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.544+0000 7f426584b640 10 monclient(hunting): picked mon.noname-c con 0x7f4260143300 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f426584b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f426584b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42600b4aa0 con 0x7f4260145410 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f426584b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f426584b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42600b4920 con 0x7f42601d7f20 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f426584b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f426584b640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42600a9960 con 0x7f4260143300 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f426584b640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:43.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f426584b640 10 monclient(hunting): authenticate will time out at 1461.707459s 2024-05-15T12:31:43.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f425e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42601d7f20 0x7f42601db420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:43.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f425e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42601d7f20 0x7f42601db420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:44884/0 (socket says 172.21.15.49:44884) 2024-05-15T12:31:43.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f425e7fc640 1 -- 172.21.15.49:0/2370440817 learned_addr learned my addr 172.21.15.49:0/2370440817 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:31:43.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f425e7fc640 10 monclient(hunting): get_auth_request con 0x7f42601d7f20 auth_method 0 2024-05-15T12:31:43.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f425e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:43.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f425e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:43.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.545+0000 7f425e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:43.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.546+0000 7f425e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:43.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.546+0000 7f425e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:43.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.546+0000 7f425e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:43.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.557+0000 7f425e7fc640 10 monclient(hunting): handle_auth_done global_id 4102 payload 824 2024-05-15T12:31:43.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.557+0000 7f425e7fc640 1 -- 172.21.15.49:0/2370440817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4260143300 msgr2=0x7f42600b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:43.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.557+0000 7f425e7fc640 1 --2- 172.21.15.49:0/2370440817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4260143300 0x7f42600b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:43.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.557+0000 7f425e7fc640 1 -- 172.21.15.49:0/2370440817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4260145410 msgr2=0x7f42601d88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:43.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.557+0000 7f425e7fc640 1 --2- 172.21.15.49:0/2370440817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4260145410 0x7f42601d88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:43.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.557+0000 7f425e7fc640 10 monclient: _finish_hunting 0 2024-05-15T12:31:43.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.557+0000 7f425e7fc640 1 monclient: found mon.noname-b 2024-05-15T12:31:43.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.557+0000 7f425e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:43.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.557+0000 7f425e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:31:43.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.557+0000 7f425e7fc640 1 -- 172.21.15.49:0/2370440817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42601d8e10 con 0x7f42601d7f20 2024-05-15T12:31:43.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.560+0000 7f425e7fc640 1 --2- 172.21.15.49:0/2370440817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42601d7f20 0x7f42601db420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f425400afd0 tx=0x7f425400cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:43.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.560+0000 7f4247fff640 10 client.?.objecter ms_handle_connect 0x7f42601d7f20 2024-05-15T12:31:43.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.560+0000 7f4247fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 1 -- 172.21.15.49:0/2370440817 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42540077e0 con 0x7f42601d7f20 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 10 monclient: dump: 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:43.555 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 10 monclient: _finish_auth 0 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 10 monclient: _check_auth_tickets 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 1 -- 172.21.15.49:0/2370440817 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425400d040 con 0x7f42601d7f20 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 1 -- 172.21.15.49:0/2370440817 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254007c00 con 0x7f42601d7f20 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f4247fff640 10 monclient: dump: 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:43.556 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f426584b640 5 monclient: authenticate success, global_id 4102 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f426584b640 10 monclient: _renew_subs 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f426584b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f426584b640 1 -- 172.21.15.49:0/2370440817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42601db960 con 0x7f42601d7f20 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f425d7fa640 4 set_mon_vals no callback set 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f426584b640 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f426584b640 10 monclient: _renew_subs 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f426584b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f426584b640 1 -- 172.21.15.49:0/2370440817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42600b7ed0 con 0x7f42601d7f20 2024-05-15T12:31:43.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.564+0000 7f426584b640 1 librados: init done 2024-05-15T12:31:43.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.565+0000 7f426584b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:43.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.565+0000 7f426584b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:43.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.565+0000 7f426584b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:43.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.565+0000 7f426584b640 1 -- 172.21.15.49:0/2370440817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426004f0e0 con 0x7f42601d7f20 2024-05-15T12:31:43.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.584+0000 7f4247fff640 1 -- 172.21.15.49:0/2370440817 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4254019020 con 0x7f42601d7f20 2024-05-15T12:31:43.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.584+0000 7f4247fff640 1 -- 172.21.15.49:0/2370440817 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4254025640 con 0x7f42601d7f20 2024-05-15T12:31:43.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.584+0000 7f4247fff640 10 client.4102.objecter ms_dispatch 0x7f4260000b90 osd_map(1..1 src has 1..1) v4 2024-05-15T12:31:43.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.584+0000 7f4247fff640 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-05-15T12:31:43.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.584+0000 7f4247fff640 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-05-15T12:31:43.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.585+0000 7f4247fff640 20 client.4102.objecter dump_active .. 0 homeless 2024-05-15T12:31:43.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.754+0000 7f4247fff640 1 -- 172.21.15.49:0/2370440817 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4254025840 con 0x7f42601d7f20 2024-05-15T12:31:43.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.754+0000 7f4247fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:43.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.754+0000 7f4247fff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:43.756 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:43.766+0000 7f83ee842040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:31:43.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.828+0000 7f426584b640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-15T12:31:43.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.828+0000 7f426584b640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-15T12:31:43.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.828+0000 7f426584b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:43.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:43.828+0000 7f426584b640 1 -- 172.21.15.49:0/2370440817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4260146ad0 con 0x7f42601d7f20 2024-05-15T12:31:43.869 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:43.880+0000 7f8a32e2d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:31:43.925 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:43.935+0000 7f83ee842040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:31:43.982 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:43.992+0000 7f83ee842040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:31:44.030 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:44.041+0000 7f8a32e2d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:31:44.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.087+0000 7f4247fff640 1 -- 172.21.15.49:0/2370440817 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4254025a20 con 0x7f42601d7f20 2024-05-15T12:31:44.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.088+0000 7f4247fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-15T12:31:44.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.088+0000 7f4247fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-05-15T12:31:44.078 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2024-05-15T12:31:44.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.088+0000 7f426584b640 10 librados: watch_flush enter 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.088+0000 7f426584b640 10 librados: watch_flush exit 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.088+0000 7f426584b640 20 client.4102.objecter shutdown clearing up homeless session... 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.088+0000 7f426584b640 10 client.4102.objecter successfully canceled tick 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 10 monclient: shutdown 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 1 -- 172.21.15.49:0/2370440817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42601d7f20 msgr2=0x7f42601db420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 1 --2- 172.21.15.49:0/2370440817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42601d7f20 0x7f42601db420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f425400afd0 tx=0x7f425400cc30 comp rx=0 tx=0).stop 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 1 -- 172.21.15.49:0/2370440817 shutdown_connections 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 1 --2- 172.21.15.49:0/2370440817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4260143300 0x7f42600b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 1 --2- 172.21.15.49:0/2370440817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42601d7f20 0x7f42601db420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 1 --2- 172.21.15.49:0/2370440817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4260145410 0x7f42601d88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 1 -- 172.21.15.49:0/2370440817 >> 172.21.15.49:0/2370440817 conn(0x7f4260066f00 msgr2=0x7f4260067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:44.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 1 -- 172.21.15.49:0/2370440817 shutdown_connections 2024-05-15T12:31:44.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 1 -- 172.21.15.49:0/2370440817 wait complete. 2024-05-15T12:31:44.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:44.089+0000 7f426584b640 1 librados: shutdown 2024-05-15T12:31:44.086 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:44.097+0000 7f8a32e2d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:31:44.090 INFO:tasks.ceph:check_enable_crimson: False 2024-05-15T12:31:44.090 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-15T12:31:44.090 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:44.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-15T12:31:44.116 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:44.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-15T12:31:44.181 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:44.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-15T12:31:44.249 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:31:44.249 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-15T12:31:44.250 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:44.259+0000 7f83ee842040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:31:44.276 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:44.276 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-15T12:31:44.302 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:44.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-15T12:31:44.303 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:44.312+0000 7f83ee842040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:31:44.360 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:44.371+0000 7f8a32e2d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:31:44.368 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:44.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-15T12:31:44.408 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:44.419+0000 7f8a32e2d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:31:44.433 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-15T12:31:44.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-15T12:31:44.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new a595cec4-7a89-4922-b2cf-adbd47129fd1 0 2024-05-15T12:31:44.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.647+0000 7fed35efa640 10 monclient: get_monmap_and_config 2024-05-15T12:31:44.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.647+0000 7fed35efa640 10 monclient: build_initial_monmap 2024-05-15T12:31:44.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.647+0000 7fed35efa640 10 monclient: monmap: 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:31:44.647940+0000 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:31:44.647940+0000 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:44.637 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:44.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.648+0000 7fed35efa640 1 Processor -- start 2024-05-15T12:31:44.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.648+0000 7fed35efa640 1 -- start start 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.648+0000 7fed35efa640 10 monclient: init 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.648+0000 7fed35efa640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.648+0000 7fed35efa640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fed300b5410 0x7fed300a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 10 monclient(hunting): picked mon.noname-a con 0x7fed300b5410 addr v2:172.21.15.49:3300/0 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed3013f990 0x7fed3013fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 10 monclient(hunting): picked mon.noname-c con 0x7fed3013f990 addr v2:172.21.15.88:3300/0 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed30146030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 10 monclient(hunting): picked mon.noname-b con 0x7fed30145c00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed300b57f0 con 0x7fed300b5410 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:44.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed300a9ea0 con 0x7fed3013f990 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed30140340 con 0x7fed30145c00 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed35efa640 10 monclient(hunting): authenticate will time out at 1477.908865s 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed2effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed3013f990 0x7fed3013fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.649+0000 7fed2ffff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed30146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2f7fe640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fed300b5410 0x7fed300a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed3013f990 0x7fed3013fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38792/0 (socket says 172.21.15.88:38792) 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2effd640 1 -- 172.21.15.88:0/1100032347 learned_addr learned my addr 172.21.15.88:0/1100032347 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2effd640 10 monclient(hunting): get_auth_request con 0x7fed3013f990 auth_method 0 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:44.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2ffff640 10 monclient(hunting): get_auth_request con 0x7fed30145c00 auth_method 0 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2f7fe640 10 monclient(hunting): get_auth_request con 0x7fed300b5410 auth_method 0 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.650+0000 7fed2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.651+0000 7fed2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.651+0000 7fed2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.651+0000 7fed2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.652+0000 7fed2effd640 10 monclient(hunting): handle_auth_done global_id 4105 payload 274 2024-05-15T12:31:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.652+0000 7fed2effd640 1 -- 172.21.15.88:0/1100032347 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 msgr2=0x7fed30146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.652+0000 7fed2effd640 1 --2- 172.21.15.88:0/1100032347 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed30146030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.652+0000 7fed2effd640 1 -- 172.21.15.88:0/1100032347 >> v2:172.21.15.49:3300/0 conn(0x7fed300b5410 msgr2=0x7fed300a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.652+0000 7fed2effd640 1 --2- 172.21.15.88:0/1100032347 >> v2:172.21.15.49:3300/0 conn(0x7fed300b5410 0x7fed300a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.652+0000 7fed2effd640 10 monclient: _finish_hunting 0 2024-05-15T12:31:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.652+0000 7fed2effd640 1 monclient: found mon.noname-c 2024-05-15T12:31:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.652+0000 7fed2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.652+0000 7fed2effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:31:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.652+0000 7fed2effd640 1 -- 172.21.15.88:0/1100032347 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed300ab470 con 0x7fed3013f990 2024-05-15T12:31:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.654+0000 7fed2effd640 1 --2- 172.21.15.88:0/1100032347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 0x7fed3013fe00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fed300ab900 tx=0x7fed1802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2285fb9e8ab41d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:44.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 1 -- 172.21.15.88:0/1100032347 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed180317c0 con 0x7fed3013f990 2024-05-15T12:31:44.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:44.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:31:44.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 10 monclient: dump: 2024-05-15T12:31:44.648 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:44.648 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:44.648 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:44.648 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:44.648 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:44.648 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 10 monclient: _finish_auth 0 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 1 -- 172.21.15.88:0/1100032347 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed18037070 con 0x7fed3013f990 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 1 -- 172.21.15.88:0/1100032347 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18031b80 con 0x7fed3013f990 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed35efa640 5 monclient: authenticate success, global_id 4105 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed35efa640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed35efa640 4 set_mon_vals no callback set 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.659+0000 7fed2e7fc640 10 monclient: dump: 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:44.650 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:44.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 10 monclient: shutdown 2024-05-15T12:31:44.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:44.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 1 -- 172.21.15.88:0/1100032347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 msgr2=0x7fed3013fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:44.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 1 --2- 172.21.15.88:0/1100032347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 0x7fed3013fe00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fed300ab900 tx=0x7fed1802b860 comp rx=0 tx=0).stop 2024-05-15T12:31:44.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 1 -- 172.21.15.88:0/1100032347 shutdown_connections 2024-05-15T12:31:44.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 1 --2- 172.21.15.88:0/1100032347 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed30146030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 1 --2- 172.21.15.88:0/1100032347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 0x7fed3013fe00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 1 --2- 172.21.15.88:0/1100032347 >> v2:172.21.15.49:3300/0 conn(0x7fed300b5410 0x7fed300a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 1 -- 172.21.15.88:0/1100032347 >> 172.21.15.88:0/1100032347 conn(0x7fed30066f00 msgr2=0x7fed30067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 1 -- 172.21.15.88:0/1100032347 shutdown_connections 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.660+0000 7fed35efa640 1 -- 172.21.15.88:0/1100032347 wait complete. 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.661+0000 7fed35efa640 10 monclient: build_initial_monmap 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.661+0000 7fed35efa640 10 monclient: monmap: 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:44.655 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.661+0000 7fed35efa640 1 librados: starting msgr at 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.661+0000 7fed35efa640 1 librados: starting objecter 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.661+0000 7fed35efa640 1 Processor -- start 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.662+0000 7fed35efa640 1 -- start start 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.662+0000 7fed35efa640 1 librados: setting wanted keys 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.662+0000 7fed35efa640 1 librados: calling monclient init 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.662+0000 7fed35efa640 10 monclient: init 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed300b5410 0x7fed300b8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 10 monclient(hunting): picked mon.noname-a con 0x7fed300b5410 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 0x7fed300b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 10 monclient(hunting): picked mon.noname-b con 0x7fed3013f990 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed301db080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:44.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 10 monclient(hunting): picked mon.noname-c con 0x7fed30145c00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed300aa5e0 con 0x7fed300b5410 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed300b4920 con 0x7fed3013f990 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed300b7830 con 0x7fed30145c00 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed35efa640 10 monclient(hunting): authenticate will time out at 1477.922881s 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed2effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 0x7fed300b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed2ffff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed301db080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed2f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed300b5410 0x7fed300b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed2effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 0x7fed300b67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38808/0 (socket says 172.21.15.88:38808) 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed2effd640 1 -- 172.21.15.88:0/3009524575 learned_addr learned my addr 172.21.15.88:0/3009524575 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed2effd640 10 monclient(hunting): get_auth_request con 0x7fed3013f990 auth_method 0 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:44.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed2effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.663+0000 7fed2effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2ffff640 10 monclient(hunting): get_auth_request con 0x7fed30145c00 auth_method 0 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2f7fe640 10 monclient(hunting): get_auth_request con 0x7fed300b5410 auth_method 0 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:44.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.664+0000 7fed2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:44.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.665+0000 7fed2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:44.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.665+0000 7fed2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:44.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.665+0000 7fed2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.667+0000 7fed2effd640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.667+0000 7fed2effd640 1 -- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 msgr2=0x7fed301db080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.667+0000 7fed2effd640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed301db080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.667+0000 7fed2effd640 1 -- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed300b5410 msgr2=0x7fed300b8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.667+0000 7fed2effd640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed300b5410 0x7fed300b8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.667+0000 7fed2effd640 10 monclient: _finish_hunting 0 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.667+0000 7fed2effd640 1 monclient: found mon.noname-b 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.667+0000 7fed2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.667+0000 7fed2effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.667+0000 7fed2effd640 1 -- 172.21.15.88:0/3009524575 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed300b7b10 con 0x7fed3013f990 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.669+0000 7fed2effd640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 0x7fed300b67b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fed18000c00 tx=0x7fed180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.669+0000 7fed27fff640 10 client.?.objecter ms_handle_connect 0x7fed3013f990 2024-05-15T12:31:44.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.669+0000 7fed27fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:44.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18032650 con 0x7fed3013f990 2024-05-15T12:31:44.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:44.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:44.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 10 monclient: dump: 2024-05-15T12:31:44.662 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:44.662 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:44.662 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 10 monclient: _finish_auth 0 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 10 monclient: _check_auth_tickets 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed180327f0 con 0x7fed3013f990 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed35efa640 5 monclient: authenticate success, global_id 4108 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed35efa640 10 monclient: _renew_subs 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed35efa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed35efa640 1 -- 172.21.15.88:0/3009524575 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed30136660 con 0x7fed3013f990 2024-05-15T12:31:44.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18032be0 con 0x7fed3013f990 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed27fff640 10 monclient: dump: 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:44.664 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:44.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed2dffb640 4 set_mon_vals no callback set 2024-05-15T12:31:44.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed35efa640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:44.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed35efa640 10 monclient: _renew_subs 2024-05-15T12:31:44.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed35efa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:44.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed35efa640 1 -- 172.21.15.88:0/3009524575 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed300b5f80 con 0x7fed3013f990 2024-05-15T12:31:44.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.673+0000 7fed35efa640 1 librados: init done 2024-05-15T12:31:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.674+0000 7fed35efa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.674+0000 7fed35efa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.674+0000 7fed35efa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.674+0000 7fed35efa640 1 -- 172.21.15.88:0/3009524575 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3004f0e0 con 0x7fed3013f990 2024-05-15T12:31:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.675+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fed1804c060 con 0x7fed3013f990 2024-05-15T12:31:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.675+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fed18006520 con 0x7fed3013f990 2024-05-15T12:31:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.675+0000 7fed27fff640 10 client.4108.objecter ms_dispatch 0x7fed30000b90 osd_map(2..2 src has 1..2) v4 2024-05-15T12:31:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.675+0000 7fed27fff640 3 client.4108.objecter handle_osd_map got epochs [2,2] > 0 2024-05-15T12:31:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.675+0000 7fed27fff640 3 client.4108.objecter handle_osd_map decoding full epoch 2 2024-05-15T12:31:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.676+0000 7fed27fff640 20 client.4108.objecter dump_active .. 0 homeless 2024-05-15T12:31:44.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.775+0000 7fed2effd640 1 -- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 msgr2=0x7fed300b67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-15T12:31:44.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.775+0000 7fed2effd640 1 -- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 msgr2=0x7fed300b67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-15T12:31:44.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed2effd640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 0x7fed300b67b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fed18000c00 tx=0x7fed180046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:31:44.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed2effd640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 0x7fed300b67b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fed18000c00 tx=0x7fed180046c0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:44.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:44.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 msgr2=0x7fed300b67b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed3013f990 0x7fed300b67b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed10006be0 0x7fed10008fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 10 monclient(hunting): picked mon.b con 0x7fed10006be0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed1000bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 10 monclient(hunting): picked mon.c con 0x7fed30145c00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed300b5410 0x7fed1000e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 10 monclient(hunting): picked mon.a con 0x7fed300b5410 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1000f000 con 0x7fed300b5410 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1000f180 con 0x7fed10006be0 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1000f380 con 0x7fed30145c00 2024-05-15T12:31:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed27fff640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.776+0000 7fed2effd640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed1000bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.777+0000 7fed2ffff640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed300b5410 0x7fed1000e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.777+0000 7fed2effd640 10 monclient(hunting): get_auth_request con 0x7fed30145c00 auth_method 0 2024-05-15T12:31:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.777+0000 7fed2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.777+0000 7fed2effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.777+0000 7fed2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.777+0000 7fed2ffff640 10 monclient(hunting): get_auth_request con 0x7fed300b5410 auth_method 0 2024-05-15T12:31:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.777+0000 7fed2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.777+0000 7fed2ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.777+0000 7fed2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:44.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.778+0000 7fed2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:44.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.778+0000 7fed2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:44.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.778+0000 7fed2effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:31:44.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.778+0000 7fed2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:44.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.778+0000 7fed2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:44.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.778+0000 7fed2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:31:44.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 10 monclient(hunting): handle_auth_done global_id 4108 payload 293 2024-05-15T12:31:44.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 1 -- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed10006be0 msgr2=0x7fed10008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:44.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed10006be0 0x7fed10008fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 1 -- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed300b5410 msgr2=0x7fed1000e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:44.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed300b5410 0x7fed1000e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:44.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 10 monclient: _finish_hunting 0 2024-05-15T12:31:44.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 1 monclient: found mon.c 2024-05-15T12:31:44.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:31:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 1 -- 172.21.15.88:0/3009524575 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fed1000f580 con 0x7fed30145c00 2024-05-15T12:31:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:31:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 1 -- 172.21.15.88:0/3009524575 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3004f0e0 con 0x7fed30145c00 2024-05-15T12:31:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 10 monclient: _finish_auth 0 2024-05-15T12:31:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 10 monclient: _check_auth_tickets 2024-05-15T12:31:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.780+0000 7fed2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:44.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.783+0000 7fed2effd640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed1000bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fed18000c00 tx=0x7fed18008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:44.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.783+0000 7fed27fff640 10 client.4108.objecter ms_handle_connect 0x7fed30145c00 2024-05-15T12:31:44.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:44.783+0000 7fed27fff640 10 client.4108.objecter resend_mon_ops 2024-05-15T12:31:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.025+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18006830 con 0x7fed30145c00 2024-05-15T12:31:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.025+0000 7fed27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.025+0000 7fed27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:31:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.025+0000 7fed27fff640 10 monclient: dump: 2024-05-15T12:31:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:45.015 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.025+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed18037020 con 0x7fed30145c00 2024-05-15T12:31:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.025+0000 7fed27fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.025+0000 7fed2d7fa640 4 set_mon_vals no callback set 2024-05-15T12:31:45.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.031+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fed1804e7e0 con 0x7fed30145c00 2024-05-15T12:31:45.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.031+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fed18040070 con 0x7fed30145c00 2024-05-15T12:31:45.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.031+0000 7fed27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:45.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.031+0000 7fed27fff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.104+0000 7fed35efa640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a595cec4-7a89-4922-b2cf-adbd47129fd1", "id": 0}] 2024-05-15T12:31:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.104+0000 7fed35efa640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a595cec4-7a89-4922-b2cf-adbd47129fd1", "id": 0}] 2024-05-15T12:31:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.104+0000 7fed35efa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:31:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.104+0000 7fed35efa640 1 -- 172.21.15.88:0/3009524575 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a595cec4-7a89-4922-b2cf-adbd47129fd1", "id": 0} v 0) v1 -- 0x7fed30146ad0 con 0x7fed30145c00 2024-05-15T12:31:45.100 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:45.110+0000 7f83ee842040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:31:45.148 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:45.159+0000 7f8a32e2d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:31:45.193 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:45.203+0000 7f83ee842040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:31:45.236 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:45.247+0000 7f8a32e2d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:31:45.242 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:45.252+0000 7f83ee842040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:31:45.281 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:45.292+0000 7f8a32e2d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:31:45.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.325+0000 7fed27fff640 1 -- 172.21.15.88:0/3009524575 <== mon.2 v2:172.21.15.49:3301/0 5 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a595cec4-7a89-4922-b2cf-adbd47129fd1", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fed18056420 con 0x7fed30145c00 2024-05-15T12:31:45.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.325+0000 7fed27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a595cec4-7a89-4922-b2cf-adbd47129fd1", "id": 0}] 2024-05-15T12:31:45.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.325+0000 7fed27fff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:45.314 INFO:teuthology.orchestra.run.smithi088.stdout:0 2024-05-15T12:31:45.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 10 librados: watch_flush enter 2024-05-15T12:31:45.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 10 librados: watch_flush exit 2024-05-15T12:31:45.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 20 client.4108.objecter shutdown clearing up homeless session... 2024-05-15T12:31:45.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 10 client.4108.objecter successfully canceled tick 2024-05-15T12:31:45.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 10 monclient: shutdown 2024-05-15T12:31:45.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:45.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 1 -- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 msgr2=0x7fed1000bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed1000bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fed18000c00 tx=0x7fed18008030 comp rx=0 tx=0).stop 2024-05-15T12:31:45.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 1 -- 172.21.15.88:0/3009524575 shutdown_connections 2024-05-15T12:31:45.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed300b5410 0x7fed1000e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed30145c00 0x7fed1000bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 1 --2- 172.21.15.88:0/3009524575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed10006be0 0x7fed10008fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 1 -- 172.21.15.88:0/3009524575 >> 172.21.15.88:0/3009524575 conn(0x7fed30066f00 msgr2=0x7fed3013ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:45.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 1 -- 172.21.15.88:0/3009524575 shutdown_connections 2024-05-15T12:31:45.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 1 -- 172.21.15.88:0/3009524575 wait complete. 2024-05-15T12:31:45.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.326+0000 7fed35efa640 1 librados: shutdown 2024-05-15T12:31:45.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new f61e8aee-68b5-4786-a307-4865e529e0cd 1 2024-05-15T12:31:45.423 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:45.433+0000 7f83ee842040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:31:45.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.437+0000 7f9a87460640 10 monclient: get_monmap_and_config 2024-05-15T12:31:45.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.437+0000 7f9a87460640 10 monclient: build_initial_monmap 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.437+0000 7f9a87460640 10 monclient: monmap: 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:31:45.437727+0000 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:31:45.437727+0000 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.437+0000 7f9a87460640 1 Processor -- start 2024-05-15T12:31:45.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 1 -- start start 2024-05-15T12:31:45.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 10 monclient: init 2024-05-15T12:31:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-05-15T12:31:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9a800b5410 0x7f9a800a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 10 monclient(hunting): picked mon.noname-c con 0x7f9a800b5410 addr v2:172.21.15.88:3300/0 2024-05-15T12:31:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a8013f990 0x7f9a8013fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 10 monclient(hunting): picked mon.noname-b con 0x7f9a8013f990 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a80145c00 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a80140310 con 0x7f9a80145c00 2024-05-15T12:31:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:45.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a800b57f0 con 0x7f9a800b5410 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800a9ea0 con 0x7f9a8013f990 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.438+0000 7f9a87460640 10 monclient(hunting): authenticate will time out at 1478.698626s 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.439+0000 7f9a851fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9a800b5410 0x7f9a800a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.439+0000 7f9a849fb640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a8013f990 0x7f9a8013fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.439+0000 7f9a859fd640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f9a80145c00 0x7f9a80146030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.88:41314/0 (socket says 172.21.15.88:41314) 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.439+0000 7f9a859fd640 1 -- 172.21.15.88:0/820389267 learned_addr learned my addr 172.21.15.88:0/820389267 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.439+0000 7f9a851fc640 10 monclient(hunting): get_auth_request con 0x7f9a800b5410 auth_method 0 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.439+0000 7f9a851fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.439+0000 7f9a851fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.439+0000 7f9a851fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:45.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.440+0000 7f9a849fb640 10 monclient(hunting): get_auth_request con 0x7f9a8013f990 auth_method 0 2024-05-15T12:31:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.440+0000 7f9a849fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.440+0000 7f9a849fb640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.440+0000 7f9a849fb640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.440+0000 7f9a851fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.440+0000 7f9a851fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.440+0000 7f9a851fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.440+0000 7f9a849fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.440+0000 7f9a849fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:45.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.440+0000 7f9a849fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.441+0000 7f9a851fc640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-05-15T12:31:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.441+0000 7f9a851fc640 1 -- 172.21.15.88:0/820389267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a8013f990 msgr2=0x7f9a8013fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.441+0000 7f9a851fc640 1 --2- 172.21.15.88:0/820389267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a8013f990 0x7f9a8013fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.441+0000 7f9a851fc640 1 -- 172.21.15.88:0/820389267 >> v1:172.21.15.49:6789/0 conn(0x7f9a80145c00 legacy=0x7f9a80146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.441+0000 7f9a851fc640 10 monclient: _finish_hunting 0 2024-05-15T12:31:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.441+0000 7f9a851fc640 1 monclient: found mon.noname-c 2024-05-15T12:31:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.441+0000 7f9a851fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.441+0000 7f9a851fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:31:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.441+0000 7f9a851fc640 1 -- 172.21.15.88:0/820389267 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a800ab470 con 0x7f9a800b5410 2024-05-15T12:31:45.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.443+0000 7f9a851fc640 1 --2- 172.21.15.88:0/820389267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a800b5410 0x7f9a800a9960 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9a740097a0 tx=0x7f9a7402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42da0c3a674a81ef server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 1 -- 172.21.15.88:0/820389267 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a740315a0 con 0x7f9a800b5410 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 10 monclient: dump: 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 10 monclient: _finish_auth 0 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 10 monclient: _check_auth_tickets 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 1 -- 172.21.15.88:0/820389267 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a74037070 con 0x7f9a800b5410 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a87460640 5 monclient: authenticate success, global_id 4111 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a87460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a87460640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a87460640 4 set_mon_vals no callback set 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 1 -- 172.21.15.88:0/820389267 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74031960 con 0x7f9a800b5410 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a73fff640 10 monclient: dump: 2024-05-15T12:31:45.437 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:45.438 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a87460640 10 monclient: shutdown 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a87460640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a87460640 1 -- 172.21.15.88:0/820389267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a800b5410 msgr2=0x7f9a800a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.447+0000 7f9a87460640 1 --2- 172.21.15.88:0/820389267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a800b5410 0x7f9a800a9960 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9a740097a0 tx=0x7f9a7402b840 comp rx=0 tx=0).stop 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.448+0000 7f9a87460640 1 -- 172.21.15.88:0/820389267 shutdown_connections 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.448+0000 7f9a87460640 1 --2- 172.21.15.88:0/820389267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a8013f990 0x7f9a8013fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.448+0000 7f9a87460640 1 --2- 172.21.15.88:0/820389267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a800b5410 0x7f9a800a9960 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.448+0000 7f9a87460640 1 -- 172.21.15.88:0/820389267 >> 172.21.15.88:0/820389267 conn(0x7f9a80066f00 msgr2=0x7f9a80067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.448+0000 7f9a87460640 1 -- 172.21.15.88:0/820389267 shutdown_connections 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.448+0000 7f9a87460640 1 -- 172.21.15.88:0/820389267 wait complete. 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.448+0000 7f9a87460640 10 monclient: build_initial_monmap 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.448+0000 7f9a87460640 10 monclient: monmap: 2024-05-15T12:31:45.442 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 librados: starting msgr at 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 librados: starting objecter 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 Processor -- start 2024-05-15T12:31:45.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 -- start start 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 librados: setting wanted keys 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 librados: calling monclient init 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient: init 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800b5410 0x7f9a80142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient(hunting): picked mon.noname-c con 0x7f9a800b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a8013f990 0x7f9a801db250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient(hunting): picked mon.noname-b con 0x7f9a8013f990 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a80145c00 0x7f9a801db790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient(hunting): picked mon.noname-a con 0x7f9a80145c00 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800b4aa0 con 0x7f9a80145c00 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800b4920 con 0x7f9a8013f990 2024-05-15T12:31:45.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8014cf30 con 0x7f9a800b5410 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.449+0000 7f9a87460640 10 monclient(hunting): authenticate will time out at 1478.709612s 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.450+0000 7f9a851fc640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800b5410 0x7f9a80142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a859fd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a80145c00 0x7f9a801db790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a849fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a8013f990 0x7f9a801db250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a851fc640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800b5410 0x7f9a80142f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.88:43134/0 (socket says 172.21.15.88:43134) 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a851fc640 1 -- 172.21.15.88:0/1401907009 learned_addr learned my addr 172.21.15.88:0/1401907009 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a851fc640 10 monclient(hunting): get_auth_request con 0x7f9a800b5410 auth_method 0 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a851fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a851fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a851fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a849fb640 10 monclient(hunting): get_auth_request con 0x7f9a8013f990 auth_method 0 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a849fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a849fb640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:45.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a849fb640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:45.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a859fd640 10 monclient(hunting): get_auth_request con 0x7f9a80145c00 auth_method 0 2024-05-15T12:31:45.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a859fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:45.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a859fd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:45.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.451+0000 7f9a859fd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:45.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.452+0000 7f9a849fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:45.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.452+0000 7f9a849fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:45.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.452+0000 7f9a849fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:45.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.452+0000 7f9a859fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:45.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.452+0000 7f9a859fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:45.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.452+0000 7f9a859fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:45.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.453+0000 7f9a851fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:45.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.453+0000 7f9a851fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:45.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.453+0000 7f9a851fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:45.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.455+0000 7f9a849fb640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-05-15T12:31:45.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.455+0000 7f9a849fb640 1 -- 172.21.15.88:0/1401907009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800b5410 msgr2=0x7f9a80142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.455+0000 7f9a849fb640 1 --2- 172.21.15.88:0/1401907009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800b5410 0x7f9a80142f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.455+0000 7f9a849fb640 1 -- 172.21.15.88:0/1401907009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a80145c00 msgr2=0x7f9a801db790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.455+0000 7f9a849fb640 1 --2- 172.21.15.88:0/1401907009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a80145c00 0x7f9a801db790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.455+0000 7f9a849fb640 10 monclient: _finish_hunting 0 2024-05-15T12:31:45.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.455+0000 7f9a849fb640 1 monclient: found mon.noname-b 2024-05-15T12:31:45.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.455+0000 7f9a849fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:45.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.455+0000 7f9a849fb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:31:45.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.455+0000 7f9a849fb640 1 -- 172.21.15.88:0/1401907009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a801424f0 con 0x7f9a8013f990 2024-05-15T12:31:45.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.457+0000 7f9a849fb640 1 --2- 172.21.15.88:0/1401907009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a8013f990 0x7f9a801db250 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00e850 tx=0x7f9a7c00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:45.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.457+0000 7f9a71ffb640 10 client.?.objecter ms_handle_connect 0x7f9a8013f990 2024-05-15T12:31:45.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.457+0000 7f9a71ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:45.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.461+0000 7f9a71ffb640 1 -- 172.21.15.88:0/1401907009 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c017070 con 0x7f9a8013f990 2024-05-15T12:31:45.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.461+0000 7f9a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:45.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.461+0000 7f9a71ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:45.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.461+0000 7f9a71ffb640 10 monclient: dump: 2024-05-15T12:31:45.453 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:45.453 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.453 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.453 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.453 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.461+0000 7f9a71ffb640 10 monclient: _finish_auth 0 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.461+0000 7f9a71ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.461+0000 7f9a71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a71ffb640 1 -- 172.21.15.88:0/1401907009 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c00c940 con 0x7f9a8013f990 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a87460640 5 monclient: authenticate success, global_id 4114 2024-05-15T12:31:45.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a87460640 10 monclient: _renew_subs 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a87460640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a87460640 1 -- 172.21.15.88:0/1401907009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a800b8590 con 0x7f9a8013f990 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a71ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a71ffb640 1 -- 172.21.15.88:0/1401907009 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c00cd10 con 0x7f9a8013f990 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a71ffb640 10 monclient: dump: 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:45.455 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a737fe640 4 set_mon_vals no callback set 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a87460640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a87460640 10 monclient: _renew_subs 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a87460640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a87460640 1 -- 172.21.15.88:0/1401907009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a80141560 con 0x7f9a8013f990 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.462+0000 7f9a87460640 1 librados: init done 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.463+0000 7f9a87460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.463+0000 7f9a87460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.463+0000 7f9a87460640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:45.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.463+0000 7f9a87460640 1 -- 172.21.15.88:0/1401907009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8004f0e0 con 0x7f9a8013f990 2024-05-15T12:31:45.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.467+0000 7f9a71ffb640 1 -- 172.21.15.88:0/1401907009 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9a7c01a070 con 0x7f9a8013f990 2024-05-15T12:31:45.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.467+0000 7f9a71ffb640 1 -- 172.21.15.88:0/1401907009 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f9a7c0065a0 con 0x7f9a8013f990 2024-05-15T12:31:45.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.467+0000 7f9a71ffb640 10 client.4114.objecter ms_dispatch 0x7f9a80000b90 osd_map(3..3 src has 1..3) v4 2024-05-15T12:31:45.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.467+0000 7f9a71ffb640 3 client.4114.objecter handle_osd_map got epochs [3,3] > 0 2024-05-15T12:31:45.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.467+0000 7f9a71ffb640 3 client.4114.objecter handle_osd_map decoding full epoch 3 2024-05-15T12:31:45.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.467+0000 7f9a71ffb640 20 client.4114.objecter dump_active .. 0 homeless 2024-05-15T12:31:45.466 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:45.478+0000 7f8a32e2d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:31:45.484 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:45.494+0000 7f83ee842040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:31:45.536 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:45.547+0000 7f8a32e2d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:31:45.539 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:45.549+0000 7f83ee842040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:31:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.587+0000 7f9a71ffb640 1 -- 172.21.15.88:0/1401907009 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9a7c0067f0 con 0x7f9a8013f990 2024-05-15T12:31:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.587+0000 7f9a71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.587+0000 7f9a71ffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:45.591 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:45.602+0000 7f8a32e2d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:31:45.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.661+0000 7f9a87460640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f61e8aee-68b5-4786-a307-4865e529e0cd", "id": 1}] 2024-05-15T12:31:45.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.661+0000 7f9a87460640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f61e8aee-68b5-4786-a307-4865e529e0cd", "id": 1}] 2024-05-15T12:31:45.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.661+0000 7f9a87460640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:45.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.661+0000 7f9a87460640 1 -- 172.21.15.88:0/1401907009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f61e8aee-68b5-4786-a307-4865e529e0cd", "id": 1} v 0) v1 -- 0x7f9a800b1fb0 con 0x7f9a8013f990 2024-05-15T12:31:45.656 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:45.666+0000 7f83ee842040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:31:45.707 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:45.718+0000 7f8a32e2d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:31:45.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.783+0000 7f9a71ffb640 1 -- 172.21.15.88:0/1401907009 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f61e8aee-68b5-4786-a307-4865e529e0cd", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9a7c005990 con 0x7f9a8013f990 2024-05-15T12:31:45.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.783+0000 7f9a71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f61e8aee-68b5-4786-a307-4865e529e0cd", "id": 1}] 2024-05-15T12:31:45.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.783+0000 7f9a71ffb640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:45.772 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 10 librados: watch_flush enter 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 10 librados: watch_flush exit 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 20 client.4114.objecter shutdown clearing up homeless session... 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 10 client.4114.objecter successfully canceled tick 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 10 monclient: shutdown 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 1 -- 172.21.15.88:0/1401907009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a8013f990 msgr2=0x7f9a801db250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 1 --2- 172.21.15.88:0/1401907009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a8013f990 0x7f9a801db250 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00e850 tx=0x7f9a7c00ed10 comp rx=0 tx=0).stop 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 1 -- 172.21.15.88:0/1401907009 shutdown_connections 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 1 --2- 172.21.15.88:0/1401907009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a80145c00 0x7f9a801db790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 1 --2- 172.21.15.88:0/1401907009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a8013f990 0x7f9a801db250 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 1 --2- 172.21.15.88:0/1401907009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800b5410 0x7f9a80142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 1 -- 172.21.15.88:0/1401907009 >> 172.21.15.88:0/1401907009 conn(0x7f9a80066f00 msgr2=0x7f9a80135b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:45.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 1 -- 172.21.15.88:0/1401907009 shutdown_connections 2024-05-15T12:31:45.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 1 -- 172.21.15.88:0/1401907009 wait complete. 2024-05-15T12:31:45.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.784+0000 7f9a87460640 1 librados: shutdown 2024-05-15T12:31:45.783 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new e3f252e2-0a0b-41a6-83be-fece689b794e 2 2024-05-15T12:31:45.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.892+0000 7f0146dc6640 10 monclient: get_monmap_and_config 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.892+0000 7f0146dc6640 10 monclient: build_initial_monmap 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.892+0000 7f0146dc6640 10 monclient: monmap: 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:31:45.893072+0000 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:31:45.893072+0000 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:45.882 INFO:teuthology.orchestra.run.smithi088.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:45.883 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.893+0000 7f0146dc6640 1 Processor -- start 2024-05-15T12:31:45.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.893+0000 7f0146dc6640 1 -- start start 2024-05-15T12:31:45.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.893+0000 7f0146dc6640 10 monclient: init 2024-05-15T12:31:45.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f01400b5410 0x7f01400a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f01400b5410 addr v2:172.21.15.49:3300/0 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f014013f990 addr v1:172.21.15.88:6789/0 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0140146310 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01400ab470 con 0x7f0140146310 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01400aa0c0 con 0x7f014013f990 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01400b57f0 con 0x7f01400b5410 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0146dc6640 10 monclient(hunting): authenticate will time out at 1479.153928s 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0137fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f014013f990 0x7f0140145c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52658/0 (socket says 172.21.15.88:52658) 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0137fff640 1 -- 172.21.15.88:0/3561273222 learned_addr learned my addr 172.21.15.88:0/3561273222 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.894+0000 7f0144b62640 1 --2- 172.21.15.88:0/3561273222 >> v2:172.21.15.49:3300/0 conn(0x7f01400b5410 0x7f01400a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:45.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.895+0000 7f0144b62640 10 monclient(hunting): get_auth_request con 0x7f01400b5410 auth_method 0 2024-05-15T12:31:45.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.895+0000 7f0144b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:45.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.895+0000 7f0144b62640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:45.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.895+0000 7f0144b62640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.896+0000 7f0144b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.896+0000 7f0144b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.896+0000 7f0144b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.897+0000 7f01377fe640 1 -- 172.21.15.88:0/3561273222 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039581088 0 0) 0x7f01400aa0c0 con 0x7f014013f990 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.897+0000 7f01377fe640 10 monclient(hunting): _negotiate 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.897+0000 7f01377fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.897+0000 7f01377fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.897+0000 7f01377fe640 10 monclient(hunting): my global_id is 4117 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.897+0000 7f01377fe640 1 -- 172.21.15.88:0/3561273222 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0128003520 con 0x7f014013f990 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.898+0000 7f0144b62640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.898+0000 7f0144b62640 1 -- 172.21.15.88:0/3561273222 >> v1:172.21.15.88:6789/0 conn(0x7f014013f990 legacy=0x7f0140145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.898+0000 7f0144b62640 1 -- 172.21.15.88:0/3561273222 >> v1:172.21.15.49:6789/0 conn(0x7f0140146310 legacy=0x7f014014cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.898+0000 7f0144b62640 10 monclient: _finish_hunting 0 2024-05-15T12:31:45.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.898+0000 7f0144b62640 1 monclient: found mon.noname-a 2024-05-15T12:31:45.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.898+0000 7f0144b62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:45.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.898+0000 7f0144b62640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:31:45.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.898+0000 7f0144b62640 1 -- 172.21.15.88:0/3561273222 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01400aa5e0 con 0x7f01400b5410 2024-05-15T12:31:45.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.905+0000 7f0144b62640 1 --2- 172.21.15.88:0/3561273222 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01400b5410 0x7f01400a9960 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f013c0097a0 tx=0x7f013c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8fea3c4c76bf0e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:45.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 1 -- 172.21.15.88:0/3561273222 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c035020 con 0x7f01400b5410 2024-05-15T12:31:45.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 10 monclient: dump: 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 10 monclient: _finish_auth 0 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 10 monclient: _check_auth_tickets 2024-05-15T12:31:45.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 1 -- 172.21.15.88:0/3561273222 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013c031830 con 0x7f01400b5410 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f0146dc6640 5 monclient: authenticate success, global_id 4125 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f0146dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f0146dc6640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 1 -- 172.21.15.88:0/3561273222 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c031c00 con 0x7f01400b5410 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f0146dc6640 4 set_mon_vals no callback set 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.909+0000 7f01377fe640 10 monclient: dump: 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:45.900 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.901 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:45.901 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:45.901 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:45.901 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.910+0000 7f0146dc6640 10 monclient: shutdown 2024-05-15T12:31:45.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.910+0000 7f0146dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:45.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.910+0000 7f0146dc6640 1 -- 172.21.15.88:0/3561273222 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01400b5410 msgr2=0x7f01400a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.910+0000 7f0146dc6640 1 --2- 172.21.15.88:0/3561273222 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01400b5410 0x7f01400a9960 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f013c0097a0 tx=0x7f013c02b8d0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.910+0000 7f0146dc6640 1 -- 172.21.15.88:0/3561273222 shutdown_connections 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.910+0000 7f0146dc6640 1 --2- 172.21.15.88:0/3561273222 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01400b5410 0x7f01400a9960 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.910+0000 7f0146dc6640 1 -- 172.21.15.88:0/3561273222 >> 172.21.15.88:0/3561273222 conn(0x7f0140066f00 msgr2=0x7f0140067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.910+0000 7f0146dc6640 1 -- 172.21.15.88:0/3561273222 shutdown_connections 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.910+0000 7f0146dc6640 1 -- 172.21.15.88:0/3561273222 wait complete. 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.911+0000 7f0146dc6640 10 monclient: build_initial_monmap 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.911+0000 7f0146dc6640 10 monclient: monmap: 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:45.905 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.911+0000 7f0146dc6640 1 librados: starting msgr at 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.911+0000 7f0146dc6640 1 librados: starting objecter 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.911+0000 7f0146dc6640 1 Processor -- start 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.911+0000 7f0146dc6640 1 -- start start 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.911+0000 7f0146dc6640 1 librados: setting wanted keys 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.911+0000 7f0146dc6640 1 librados: calling monclient init 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.911+0000 7f0146dc6640 10 monclient: init 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01400b5410 0x7f0140145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 10 monclient(hunting): picked mon.noname-b con 0x7f01400b5410 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f014013f990 0x7f01401d8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:45.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 10 monclient(hunting): picked mon.noname-c con 0x7f014013f990 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0140146310 0x7f0140142870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f0140146310 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014013fe00 con 0x7f0140146310 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01400a9ea0 con 0x7f01400b5410 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014013ff80 con 0x7f014013f990 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0146dc6640 10 monclient(hunting): authenticate will time out at 1479.172253s 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.912+0000 7f0144b62640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01400b5410 0x7f0140145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0145363640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0140146310 0x7f0140142870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0137fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f014013f990 0x7f01401d8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:45.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0144b62640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01400b5410 0x7f0140145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38850/0 (socket says 172.21.15.88:38850) 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0144b62640 1 -- 172.21.15.88:0/572498154 learned_addr learned my addr 172.21.15.88:0/572498154 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0144b62640 10 monclient(hunting): get_auth_request con 0x7f01400b5410 auth_method 0 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0144b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0144b62640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0144b62640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0137fff640 10 monclient(hunting): get_auth_request con 0x7f014013f990 auth_method 0 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0137fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0137fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0137fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0145363640 10 monclient(hunting): get_auth_request con 0x7f0140146310 auth_method 0 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0145363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0145363640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.913+0000 7f0145363640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.914+0000 7f0144b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.914+0000 7f0144b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:45.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.914+0000 7f0144b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:45.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.914+0000 7f0137fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:45.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.914+0000 7f0137fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:45.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.914+0000 7f0137fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:45.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.914+0000 7f0145363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:45.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.914+0000 7f0145363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:45.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.914+0000 7f0145363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:45.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.916+0000 7f0144b62640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-05-15T12:31:45.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.916+0000 7f0144b62640 1 -- 172.21.15.88:0/572498154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f014013f990 msgr2=0x7f01401d8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.916+0000 7f0144b62640 1 --2- 172.21.15.88:0/572498154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f014013f990 0x7f01401d8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.916+0000 7f0144b62640 1 -- 172.21.15.88:0/572498154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0140146310 msgr2=0x7f0140142870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:45.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.916+0000 7f0144b62640 1 --2- 172.21.15.88:0/572498154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0140146310 0x7f0140142870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:45.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.917+0000 7f0144b62640 10 monclient: _finish_hunting 0 2024-05-15T12:31:45.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.917+0000 7f0144b62640 1 monclient: found mon.noname-b 2024-05-15T12:31:45.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.917+0000 7f0144b62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:45.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.917+0000 7f0144b62640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:31:45.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.917+0000 7f0144b62640 1 -- 172.21.15.88:0/572498154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01401d9050 con 0x7f01400b5410 2024-05-15T12:31:45.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.919+0000 7f0144b62640 1 --2- 172.21.15.88:0/572498154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01400b5410 0x7f0140145660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f013c02bee0 tx=0x7f013c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:45.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.919+0000 7f01357fa640 10 client.?.objecter ms_handle_connect 0x7f01400b5410 2024-05-15T12:31:45.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.919+0000 7f01357fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 1 -- 172.21.15.88:0/572498154 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c041070 con 0x7f01400b5410 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 10 monclient: dump: 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.912 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 10 monclient: _finish_auth 0 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 10 monclient: _check_auth_tickets 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 1 -- 172.21.15.88:0/572498154 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013c03b920 con 0x7f01400b5410 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 1 -- 172.21.15.88:0/572498154 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c035040 con 0x7f01400b5410 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f0146dc6640 5 monclient: authenticate success, global_id 4120 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f0146dc6640 10 monclient: _renew_subs 2024-05-15T12:31:45.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f0146dc6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:45.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f0146dc6640 1 -- 172.21.15.88:0/572498154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0140144ed0 con 0x7f01400b5410 2024-05-15T12:31:45.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f01357fa640 10 monclient: dump: 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f0136ffd640 4 set_mon_vals no callback set 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f0146dc6640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:45.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f0146dc6640 10 monclient: _renew_subs 2024-05-15T12:31:45.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f0146dc6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:45.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f0146dc6640 1 -- 172.21.15.88:0/572498154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0140142db0 con 0x7f01400b5410 2024-05-15T12:31:45.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.922+0000 7f0146dc6640 1 librados: init done 2024-05-15T12:31:45.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.923+0000 7f0146dc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:45.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.923+0000 7f0146dc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:45.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.923+0000 7f0146dc6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:45.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:45.923+0000 7f0146dc6640 1 -- 172.21.15.88:0/572498154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014004f0e0 con 0x7f01400b5410 2024-05-15T12:31:46.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.026+0000 7f01357fa640 1 -- 172.21.15.88:0/572498154 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f013c04c060 con 0x7f01400b5410 2024-05-15T12:31:46.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.026+0000 7f01357fa640 1 -- 172.21.15.88:0/572498154 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f013c03a860 con 0x7f01400b5410 2024-05-15T12:31:46.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.026+0000 7f01357fa640 10 client.4120.objecter ms_dispatch 0x7f0140000b90 osd_map(4..4 src has 1..4) v4 2024-05-15T12:31:46.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.026+0000 7f01357fa640 3 client.4120.objecter handle_osd_map got epochs [4,4] > 0 2024-05-15T12:31:46.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.026+0000 7f01357fa640 3 client.4120.objecter handle_osd_map decoding full epoch 4 2024-05-15T12:31:46.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.026+0000 7f01357fa640 20 client.4120.objecter dump_active .. 0 homeless 2024-05-15T12:31:46.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.026+0000 7f01357fa640 1 -- 172.21.15.88:0/572498154 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f013c03ab70 con 0x7f01400b5410 2024-05-15T12:31:46.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.026+0000 7f01357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:46.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.026+0000 7f01357fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:46.064 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:46.074+0000 7f83ee842040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:31:46.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.099+0000 7f0146dc6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e3f252e2-0a0b-41a6-83be-fece689b794e", "id": 2}] 2024-05-15T12:31:46.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.099+0000 7f0146dc6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e3f252e2-0a0b-41a6-83be-fece689b794e", "id": 2}] 2024-05-15T12:31:46.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.099+0000 7f0146dc6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:46.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.099+0000 7f0146dc6640 1 -- 172.21.15.88:0/572498154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e3f252e2-0a0b-41a6-83be-fece689b794e", "id": 2} v 0) v1 -- 0x7f0140146ad0 con 0x7f01400b5410 2024-05-15T12:31:46.104 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:46.115+0000 7f8a32e2d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:31:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.190+0000 7f01357fa640 1 -- 172.21.15.88:0/572498154 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e3f252e2-0a0b-41a6-83be-fece689b794e", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f013c03bdf0 con 0x7f01400b5410 2024-05-15T12:31:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.190+0000 7f01357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e3f252e2-0a0b-41a6-83be-fece689b794e", "id": 2}] 2024-05-15T12:31:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.190+0000 7f01357fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:46.180 INFO:teuthology.orchestra.run.smithi088.stdout:2 2024-05-15T12:31:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.191+0000 7f0146dc6640 10 librados: watch_flush enter 2024-05-15T12:31:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.191+0000 7f0146dc6640 10 librados: watch_flush exit 2024-05-15T12:31:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.191+0000 7f0146dc6640 20 client.4120.objecter shutdown clearing up homeless session... 2024-05-15T12:31:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.191+0000 7f0146dc6640 10 client.4120.objecter successfully canceled tick 2024-05-15T12:31:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 10 monclient: shutdown 2024-05-15T12:31:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 1 -- 172.21.15.88:0/572498154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01400b5410 msgr2=0x7f0140145660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 1 --2- 172.21.15.88:0/572498154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01400b5410 0x7f0140145660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f013c02bee0 tx=0x7f013c0046c0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 1 -- 172.21.15.88:0/572498154 shutdown_connections 2024-05-15T12:31:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 1 --2- 172.21.15.88:0/572498154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0140146310 0x7f0140142870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 1 --2- 172.21.15.88:0/572498154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f014013f990 0x7f01401d8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 1 --2- 172.21.15.88:0/572498154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01400b5410 0x7f0140145660 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 1 -- 172.21.15.88:0/572498154 >> 172.21.15.88:0/572498154 conn(0x7f0140066f00 msgr2=0x7f014013a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 1 -- 172.21.15.88:0/572498154 shutdown_connections 2024-05-15T12:31:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 1 -- 172.21.15.88:0/572498154 wait complete. 2024-05-15T12:31:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.192+0000 7f0146dc6640 1 librados: shutdown 2024-05-15T12:31:46.191 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a 3 2024-05-15T12:31:46.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.300+0000 7fb328c7e640 10 monclient: get_monmap_and_config 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.300+0000 7fb328c7e640 10 monclient: build_initial_monmap 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.300+0000 7fb328c7e640 10 monclient: monmap: 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:31:46.300949+0000 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:31:46.300949+0000 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:46.290 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 1 Processor -- start 2024-05-15T12:31:46.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 1 -- start start 2024-05-15T12:31:46.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 10 monclient: init 2024-05-15T12:31:46.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:46.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-05-15T12:31:46.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3240b5410 addr v1:172.21.15.88:6789/0 2024-05-15T12:31:46.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3241400a0 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:46.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb324146310 0x7fb3240a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:46.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 10 monclient(hunting): picked mon.noname-c con 0x7fb324146310 addr v2:172.21.15.88:3300/0 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3240ab470 con 0x7fb3241400a0 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.301+0000 7fb328c7e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3240aa0c0 con 0x7fb3240b5410 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.302+0000 7fb328c7e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.302+0000 7fb328c7e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3240b57f0 con 0x7fb324146310 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.302+0000 7fb328c7e640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.302+0000 7fb328c7e640 10 monclient(hunting): authenticate will time out at 1479.561784s 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.302+0000 7fb321d9b640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fb3241400a0 0x7fb324145c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.88:41330/0 (socket says 172.21.15.88:41330) 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.302+0000 7fb321d9b640 1 -- 172.21.15.88:0/334286397 learned_addr learned my addr 172.21.15.88:0/334286397 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.302+0000 7fb322d9d640 1 --2- 172.21.15.88:0/334286397 >> v2:172.21.15.88:3300/0 conn(0x7fb324146310 0x7fb3240a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.303+0000 7fb322d9d640 10 monclient(hunting): get_auth_request con 0x7fb324146310 auth_method 0 2024-05-15T12:31:46.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.303+0000 7fb322d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:46.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.303+0000 7fb322d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:46.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.303+0000 7fb322d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.304+0000 7fb322d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.304+0000 7fb322d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.304+0000 7fb322d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.305+0000 7fb32159a640 1 -- 172.21.15.88:0/334286397 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854509577 0 0) 0x7fb3240ab470 con 0x7fb3241400a0 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.305+0000 7fb32159a640 10 monclient(hunting): _negotiate 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.305+0000 7fb32159a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.305+0000 7fb32159a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.305+0000 7fb32159a640 10 monclient(hunting): my global_id is 4134 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.305+0000 7fb32159a640 1 -- 172.21.15.88:0/334286397 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb30c003520 con 0x7fb3241400a0 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.306+0000 7fb322d9d640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-05-15T12:31:46.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.306+0000 7fb322d9d640 1 -- 172.21.15.88:0/334286397 >> v1:172.21.15.88:6789/0 conn(0x7fb3240b5410 legacy=0x7fb32413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.306+0000 7fb322d9d640 1 -- 172.21.15.88:0/334286397 >> v1:172.21.15.49:6789/0 conn(0x7fb3241400a0 legacy=0x7fb324145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.306+0000 7fb322d9d640 10 monclient: _finish_hunting 0 2024-05-15T12:31:46.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.306+0000 7fb322d9d640 1 monclient: found mon.noname-c 2024-05-15T12:31:46.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.306+0000 7fb322d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:46.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.306+0000 7fb322d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:31:46.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.306+0000 7fb322d9d640 1 -- 172.21.15.88:0/334286397 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3240aa5e0 con 0x7fb324146310 2024-05-15T12:31:46.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.310+0000 7fb322d9d640 1 --2- 172.21.15.88:0/334286397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324146310 0x7fb3240a9960 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009d90 tx=0x7fb31c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e14daf9be164f4f server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 1 -- 172.21.15.88:0/334286397 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c039070 con 0x7fb324146310 2024-05-15T12:31:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:31:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 10 monclient: dump: 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 10 monclient: _finish_auth 0 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 10 monclient: _check_auth_tickets 2024-05-15T12:31:46.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 1 -- 172.21.15.88:0/334286397 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c02cc60 con 0x7fb324146310 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 1 -- 172.21.15.88:0/334286397 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c034660 con 0x7fb324146310 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb328c7e640 5 monclient: authenticate success, global_id 4123 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb328c7e640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb328c7e640 4 set_mon_vals no callback set 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.314+0000 7fb32159a640 10 monclient: dump: 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:46.305 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:46.306 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:46.306 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.315+0000 7fb328c7e640 10 monclient: shutdown 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.315+0000 7fb328c7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.315+0000 7fb328c7e640 1 -- 172.21.15.88:0/334286397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324146310 msgr2=0x7fb3240a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.315+0000 7fb328c7e640 1 --2- 172.21.15.88:0/334286397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324146310 0x7fb3240a9960 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009d90 tx=0x7fb31c02beb0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.315+0000 7fb328c7e640 1 -- 172.21.15.88:0/334286397 shutdown_connections 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.315+0000 7fb328c7e640 1 --2- 172.21.15.88:0/334286397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324146310 0x7fb3240a9960 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.315+0000 7fb328c7e640 1 -- 172.21.15.88:0/334286397 >> 172.21.15.88:0/334286397 conn(0x7fb324066f00 msgr2=0x7fb324067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.315+0000 7fb328c7e640 1 -- 172.21.15.88:0/334286397 shutdown_connections 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.315+0000 7fb328c7e640 1 -- 172.21.15.88:0/334286397 wait complete. 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.316+0000 7fb328c7e640 10 monclient: build_initial_monmap 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.316+0000 7fb328c7e640 10 monclient: monmap: 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.317+0000 7fb328c7e640 1 librados: starting msgr at 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.317+0000 7fb328c7e640 1 librados: starting objecter 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.317+0000 7fb328c7e640 1 Processor -- start 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.317+0000 7fb328c7e640 1 -- start start 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.317+0000 7fb328c7e640 1 librados: setting wanted keys 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.317+0000 7fb328c7e640 1 librados: calling monclient init 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.317+0000 7fb328c7e640 10 monclient: init 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3240b5410 0x7fb324145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 10 monclient(hunting): picked mon.noname-c con 0x7fb3240b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3241400a0 0x7fb3241314b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 10 monclient(hunting): picked mon.noname-a con 0x7fb3241400a0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324146310 0x7fb3241319f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 10 monclient(hunting): picked mon.noname-b con 0x7fb324146310 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32414df40 con 0x7fb3241400a0 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3240a9ea0 con 0x7fb324146310 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32414e0c0 con 0x7fb3240b5410 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb328c7e640 10 monclient(hunting): authenticate will time out at 1479.578089s 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb322d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324146310 0x7fb3241319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:46.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.318+0000 7fb32259c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3240b5410 0x7fb324145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb321d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3241400a0 0x7fb3241314b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb32259c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3240b5410 0x7fb324145500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.88:43158/0 (socket says 172.21.15.88:43158) 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb32259c640 1 -- 172.21.15.88:0/4150173598 learned_addr learned my addr 172.21.15.88:0/4150173598 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb32259c640 10 monclient(hunting): get_auth_request con 0x7fb3240b5410 auth_method 0 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb32259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb32259c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb32259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb322d9d640 10 monclient(hunting): get_auth_request con 0x7fb324146310 auth_method 0 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb322d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb322d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb322d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb321d9b640 10 monclient(hunting): get_auth_request con 0x7fb3241400a0 auth_method 0 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb321d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb321d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb321d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb322d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb322d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb322d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb321d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb321d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.319+0000 7fb321d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.320+0000 7fb32259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.320+0000 7fb32259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.320+0000 7fb32259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.322+0000 7fb322d9d640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-05-15T12:31:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.322+0000 7fb322d9d640 1 -- 172.21.15.88:0/4150173598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3240b5410 msgr2=0x7fb324145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.322+0000 7fb322d9d640 1 --2- 172.21.15.88:0/4150173598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3240b5410 0x7fb324145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.322+0000 7fb322d9d640 1 -- 172.21.15.88:0/4150173598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3241400a0 msgr2=0x7fb3241314b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.322+0000 7fb322d9d640 1 --2- 172.21.15.88:0/4150173598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3241400a0 0x7fb3241314b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.322+0000 7fb322d9d640 10 monclient: _finish_hunting 0 2024-05-15T12:31:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.322+0000 7fb322d9d640 1 monclient: found mon.noname-b 2024-05-15T12:31:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.322+0000 7fb322d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:46.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.322+0000 7fb322d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:31:46.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.322+0000 7fb322d9d640 1 -- 172.21.15.88:0/4150173598 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb324141010 con 0x7fb324146310 2024-05-15T12:31:46.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.324+0000 7fb322d9d640 1 --2- 172.21.15.88:0/4150173598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324146310 0x7fb3241319f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb31c033000 tx=0x7fb31c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:46.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.324+0000 7fb312ffd640 10 client.?.objecter ms_handle_connect 0x7fb324146310 2024-05-15T12:31:46.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.324+0000 7fb312ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 1 -- 172.21.15.88:0/4150173598 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c044020 con 0x7fb324146310 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 10 monclient: dump: 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 10 monclient: _finish_auth 0 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 1 -- 172.21.15.88:0/4150173598 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c03f470 con 0x7fb324146310 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 1 -- 172.21.15.88:0/4150173598 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c039040 con 0x7fb324146310 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb328c7e640 5 monclient: authenticate success, global_id 4129 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb328c7e640 10 monclient: _renew_subs 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb328c7e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb328c7e640 1 -- 172.21.15.88:0/4150173598 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3241412f0 con 0x7fb324146310 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb312ffd640 10 monclient: dump: 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb320d99640 4 set_mon_vals no callback set 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb328c7e640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb328c7e640 10 monclient: _renew_subs 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb328c7e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb328c7e640 1 -- 172.21.15.88:0/4150173598 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb324131f30 con 0x7fb324146310 2024-05-15T12:31:46.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.328+0000 7fb328c7e640 1 librados: init done 2024-05-15T12:31:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.329+0000 7fb328c7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.329+0000 7fb328c7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.329+0000 7fb328c7e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.329+0000 7fb328c7e640 1 -- 172.21.15.88:0/4150173598 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32404f0e0 con 0x7fb324146310 2024-05-15T12:31:46.338 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:46.348+0000 7f83ee842040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:31:46.378 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:46.389+0000 7f8a32e2d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:31:46.392 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:46.403+0000 7f83ee842040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:31:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.434+0000 7fb312ffd640 1 -- 172.21.15.88:0/4150173598 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb31c047020 con 0x7fb324146310 2024-05-15T12:31:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.434+0000 7fb312ffd640 1 -- 172.21.15.88:0/4150173598 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fb31c004490 con 0x7fb324146310 2024-05-15T12:31:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.434+0000 7fb312ffd640 10 client.4129.objecter ms_dispatch 0x7fb324000b90 osd_map(5..5 src has 1..5) v4 2024-05-15T12:31:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.434+0000 7fb312ffd640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-05-15T12:31:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.434+0000 7fb312ffd640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-05-15T12:31:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.434+0000 7fb312ffd640 20 client.4129.objecter dump_active .. 0 homeless 2024-05-15T12:31:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.434+0000 7fb312ffd640 1 -- 172.21.15.88:0/4150173598 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb31c04e070 con 0x7fb324146310 2024-05-15T12:31:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.434+0000 7fb312ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.434+0000 7fb312ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:46.435 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:46.446+0000 7f8a32e2d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:31:46.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.508+0000 7fb328c7e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a", "id": 3}] 2024-05-15T12:31:46.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.508+0000 7fb328c7e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a", "id": 3}] 2024-05-15T12:31:46.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.508+0000 7fb328c7e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:46.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.508+0000 7fb328c7e640 1 -- 172.21.15.88:0/4150173598 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a", "id": 3} v 0) v1 -- 0x7fb324146ad0 con 0x7fb324146310 2024-05-15T12:31:46.516 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:46.526+0000 7f83ee842040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:31:46.558 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:46.569+0000 7f8a32e2d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:31:46.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.582+0000 7fb312ffd640 1 -- 172.21.15.88:0/4150173598 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb31c03fe20 con 0x7fb324146310 2024-05-15T12:31:46.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.582+0000 7fb312ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a", "id": 3}] 2024-05-15T12:31:46.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.582+0000 7fb312ffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:46.571 INFO:teuthology.orchestra.run.smithi088.stdout:3 2024-05-15T12:31:46.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 10 librados: watch_flush enter 2024-05-15T12:31:46.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 10 librados: watch_flush exit 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 20 client.4129.objecter shutdown clearing up homeless session... 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 10 client.4129.objecter successfully canceled tick 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 10 monclient: shutdown 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 1 -- 172.21.15.88:0/4150173598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324146310 msgr2=0x7fb3241319f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 1 --2- 172.21.15.88:0/4150173598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324146310 0x7fb3241319f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb31c033000 tx=0x7fb31c004a40 comp rx=0 tx=0).stop 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 1 -- 172.21.15.88:0/4150173598 shutdown_connections 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 1 --2- 172.21.15.88:0/4150173598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324146310 0x7fb3241319f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 1 --2- 172.21.15.88:0/4150173598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3241400a0 0x7fb3241314b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 1 --2- 172.21.15.88:0/4150173598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3240b5410 0x7fb324145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 1 -- 172.21.15.88:0/4150173598 >> 172.21.15.88:0/4150173598 conn(0x7fb324066f00 msgr2=0x7fb32413a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 1 -- 172.21.15.88:0/4150173598 shutdown_connections 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 1 -- 172.21.15.88:0/4150173598 wait complete. 2024-05-15T12:31:46.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.583+0000 7fb328c7e640 1 librados: shutdown 2024-05-15T12:31:46.586 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 9be5d412-2421-4c86-8d81-540b24dc2616 4 2024-05-15T12:31:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.695+0000 7fde77bf1640 10 monclient: get_monmap_and_config 2024-05-15T12:31:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.695+0000 7fde77bf1640 10 monclient: build_initial_monmap 2024-05-15T12:31:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.695+0000 7fde77bf1640 10 monclient: monmap: 2024-05-15T12:31:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:31:46.695484+0000 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:31:46.695484+0000 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.695+0000 7fde77bf1640 1 Processor -- start 2024-05-15T12:31:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.695+0000 7fde77bf1640 1 -- start start 2024-05-15T12:31:46.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.695+0000 7fde77bf1640 10 monclient: init 2024-05-15T12:31:46.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:46.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-05-15T12:31:46.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fde700b5410 0x7fde700a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:46.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 10 monclient(hunting): picked mon.noname-a con 0x7fde700b5410 addr v2:172.21.15.49:3300/0 2024-05-15T12:31:46.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde7013f990 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde70146310 0x7fde7013fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 10 monclient(hunting): picked mon.noname-b con 0x7fde70146310 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde700ab470 con 0x7fde7013f990 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde700b57f0 con 0x7fde700b5410 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde700a9ea0 con 0x7fde70146310 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.696+0000 7fde77bf1640 10 monclient(hunting): authenticate will time out at 1479.956435s 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7598d640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fde700b5410 0x7fde700a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7618e640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde70146310 0x7fde7013fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7618e640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde70146310 0x7fde7013fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.88:43164/0 (socket says 172.21.15.88:43164) 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7618e640 1 -- 172.21.15.88:0/4105909406 learned_addr learned my addr 172.21.15.88:0/4105909406 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7618e640 10 monclient(hunting): get_auth_request con 0x7fde70146310 auth_method 0 2024-05-15T12:31:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7618e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7618e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7618e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7598d640 10 monclient(hunting): get_auth_request con 0x7fde700b5410 auth_method 0 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7598d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7598d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.697+0000 7fde7598d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.698+0000 7fde7618e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.698+0000 7fde7618e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.698+0000 7fde7618e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.698+0000 7fde7598d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.698+0000 7fde7598d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.698+0000 7fde7598d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.700+0000 7fde7618e640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2024-05-15T12:31:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.700+0000 7fde7618e640 1 -- 172.21.15.88:0/4105909406 >> v2:172.21.15.49:3300/0 conn(0x7fde700b5410 msgr2=0x7fde700a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.700+0000 7fde7618e640 1 --2- 172.21.15.88:0/4105909406 >> v2:172.21.15.49:3300/0 conn(0x7fde700b5410 0x7fde700a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.700+0000 7fde7618e640 1 -- 172.21.15.88:0/4105909406 >> v1:172.21.15.49:6789/0 conn(0x7fde7013f990 legacy=0x7fde70145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.700+0000 7fde7618e640 10 monclient: _finish_hunting 0 2024-05-15T12:31:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.700+0000 7fde7618e640 1 monclient: found mon.noname-b 2024-05-15T12:31:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.700+0000 7fde7618e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.700+0000 7fde7618e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:31:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.700+0000 7fde7618e640 1 -- 172.21.15.88:0/4105909406 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde700aa0c0 con 0x7fde70146310 2024-05-15T12:31:46.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.702+0000 7fde7618e640 1 --2- 172.21.15.88:0/4105909406 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde70146310 0x7fde7013fe00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde68009e50 tx=0x7fde6802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67ef9dc4a900db28 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:46.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.705+0000 7fde7498b640 1 -- 172.21.15.88:0/4105909406 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6802ca70 con 0x7fde70146310 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.705+0000 7fde7498b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.705+0000 7fde7498b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.705+0000 7fde7498b640 10 monclient: dump: 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde7498b640 10 monclient: _finish_auth 0 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde7498b640 10 monclient: _check_auth_tickets 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde7498b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde7498b640 1 -- 172.21.15.88:0/4105909406 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6802cc10 con 0x7fde70146310 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde7498b640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde7498b640 1 -- 172.21.15.88:0/4105909406 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde680343f0 con 0x7fde70146310 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde7498b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde7498b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde7498b640 10 monclient: dump: 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:46.697 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 5 monclient: authenticate success, global_id 4121 2024-05-15T12:31:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 4 set_mon_vals no callback set 2024-05-15T12:31:46.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 10 monclient: shutdown 2024-05-15T12:31:46.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:46.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 1 -- 172.21.15.88:0/4105909406 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde70146310 msgr2=0x7fde7013fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 1 --2- 172.21.15.88:0/4105909406 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde70146310 0x7fde7013fe00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde68009e50 tx=0x7fde6802bed0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 1 -- 172.21.15.88:0/4105909406 shutdown_connections 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 1 --2- 172.21.15.88:0/4105909406 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde70146310 0x7fde7013fe00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 1 --2- 172.21.15.88:0/4105909406 >> v2:172.21.15.49:3300/0 conn(0x7fde700b5410 0x7fde700a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 1 -- 172.21.15.88:0/4105909406 >> 172.21.15.88:0/4105909406 conn(0x7fde70066f00 msgr2=0x7fde70067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 1 -- 172.21.15.88:0/4105909406 shutdown_connections 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.706+0000 7fde77bf1640 1 -- 172.21.15.88:0/4105909406 wait complete. 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.707+0000 7fde77bf1640 10 monclient: build_initial_monmap 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.707+0000 7fde77bf1640 10 monclient: monmap: 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.707+0000 7fde77bf1640 1 librados: starting msgr at 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.707+0000 7fde77bf1640 1 librados: starting objecter 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.707+0000 7fde77bf1640 1 Processor -- start 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.707+0000 7fde77bf1640 1 -- start start 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.707+0000 7fde77bf1640 1 librados: setting wanted keys 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.707+0000 7fde77bf1640 1 librados: calling monclient init 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.707+0000 7fde77bf1640 10 monclient: init 2024-05-15T12:31:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde700b5410 0x7fde70144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 10 monclient(hunting): picked mon.noname-b con 0x7fde700b5410 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7013f990 0x7fde701d8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 10 monclient(hunting): picked mon.noname-c con 0x7fde7013f990 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde70146310 0x7fde701d8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 10 monclient(hunting): picked mon.noname-a con 0x7fde70146310 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde700b4920 con 0x7fde70146310 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde70140340 con 0x7fde700b5410 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde700b4aa0 con 0x7fde7013f990 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde77bf1640 10 monclient(hunting): authenticate will time out at 1479.968130s 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde7618e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde70146310 0x7fde701d8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde7518c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7013f990 0x7fde701d8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.708+0000 7fde7598d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde700b5410 0x7fde70144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7518c640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7013f990 msgr2=0x7fde701d8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7518c640 1 -- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7013f990 msgr2=0x7fde701d8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7518c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7013f990 0x7fde701d8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7518c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7013f990 0x7fde701d8570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7618e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde70146310 0x7fde701d8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.88:57932/0 (socket says 172.21.15.88:57932) 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7618e640 1 -- 172.21.15.88:0/2915062009 learned_addr learned my addr 172.21.15.88:0/2915062009 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7618e640 10 monclient(hunting): get_auth_request con 0x7fde70146310 auth_method 0 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7618e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7618e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7618e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7598d640 10 monclient(hunting): get_auth_request con 0x7fde700b5410 auth_method 0 2024-05-15T12:31:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7598d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7598d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7598d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7618e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7618e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7618e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.709+0000 7fde7598d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.710+0000 7fde7598d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.710+0000 7fde7598d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.712+0000 7fde7618e640 10 monclient(hunting): handle_auth_done global_id 4143 payload 824 2024-05-15T12:31:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.712+0000 7fde7618e640 1 -- 172.21.15.88:0/2915062009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7013f990 msgr2=0x7fde701d8570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-15T12:31:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.712+0000 7fde7618e640 1 --2- 172.21.15.88:0/2915062009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7013f990 0x7fde701d8570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.712+0000 7fde7618e640 1 -- 172.21.15.88:0/2915062009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde700b5410 msgr2=0x7fde70144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.712+0000 7fde7618e640 1 --2- 172.21.15.88:0/2915062009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde700b5410 0x7fde70144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.712+0000 7fde7618e640 10 monclient: _finish_hunting 0 2024-05-15T12:31:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.712+0000 7fde7618e640 1 monclient: found mon.noname-a 2024-05-15T12:31:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.712+0000 7fde7618e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.712+0000 7fde7618e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:31:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.712+0000 7fde7618e640 1 -- 172.21.15.88:0/2915062009 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde70140e80 con 0x7fde70146310 2024-05-15T12:31:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.714+0000 7fde7618e640 1 --2- 172.21.15.88:0/2915062009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde70146310 0x7fde701d8ab0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde68036040 tx=0x7fde68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.714+0000 7fde627fc640 10 client.?.objecter ms_handle_connect 0x7fde70146310 2024-05-15T12:31:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.714+0000 7fde627fc640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:46.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 1 -- 172.21.15.88:0/2915062009 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68043020 con 0x7fde70146310 2024-05-15T12:31:46.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 10 monclient: dump: 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 10 monclient: _finish_auth 0 2024-05-15T12:31:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 10 monclient: _check_auth_tickets 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 1 -- 172.21.15.88:0/2915062009 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6803e450 con 0x7fde70146310 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde77bf1640 5 monclient: authenticate success, global_id 4143 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde77bf1640 10 monclient: _renew_subs 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde77bf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde77bf1640 1 -- 172.21.15.88:0/2915062009 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde701450e0 con 0x7fde70146310 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 1 -- 172.21.15.88:0/2915062009 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68009510 con 0x7fde70146310 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde627fc640 10 monclient: dump: 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde63fff640 4 set_mon_vals no callback set 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde77bf1640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde77bf1640 10 monclient: _renew_subs 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde77bf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde77bf1640 1 -- 172.21.15.88:0/2915062009 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde70140500 con 0x7fde70146310 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.718+0000 7fde77bf1640 1 librados: init done 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.719+0000 7fde77bf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.719+0000 7fde77bf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.719+0000 7fde77bf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:31:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.719+0000 7fde77bf1640 1 -- 172.21.15.88:0/2915062009 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7004f0e0 con 0x7fde70146310 2024-05-15T12:31:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.723+0000 7fde627fc640 1 -- 172.21.15.88:0/2915062009 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fde68046020 con 0x7fde70146310 2024-05-15T12:31:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.723+0000 7fde627fc640 1 -- 172.21.15.88:0/2915062009 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fde68051940 con 0x7fde70146310 2024-05-15T12:31:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.723+0000 7fde627fc640 10 client.4143.objecter ms_dispatch 0x7fde70000b90 osd_map(6..6 src has 1..6) v4 2024-05-15T12:31:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.723+0000 7fde627fc640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-05-15T12:31:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.723+0000 7fde627fc640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-05-15T12:31:46.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.723+0000 7fde627fc640 20 client.4143.objecter dump_active .. 0 homeless 2024-05-15T12:31:46.822 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:46.832+0000 7f83ee842040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:31:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.845+0000 7fde627fc640 1 -- 172.21.15.88:0/2915062009 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fde6804d030 con 0x7fde70146310 2024-05-15T12:31:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.845+0000 7fde627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.845+0000 7fde627fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:46.868 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:46.879+0000 7f8a32e2d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:31:46.873 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:46.883+0000 7f83ee842040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:31:46.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.916+0000 7fde77bf1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9be5d412-2421-4c86-8d81-540b24dc2616", "id": 4}] 2024-05-15T12:31:46.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.916+0000 7fde77bf1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9be5d412-2421-4c86-8d81-540b24dc2616", "id": 4}] 2024-05-15T12:31:46.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.916+0000 7fde77bf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:31:46.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.916+0000 7fde77bf1640 1 -- 172.21.15.88:0/2915062009 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9be5d412-2421-4c86-8d81-540b24dc2616", "id": 4} v 0) v1 -- 0x7fde7013d6c0 con 0x7fde70146310 2024-05-15T12:31:46.915 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:46.926+0000 7f8a32e2d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:31:46.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.961+0000 7fde627fc640 1 -- 172.21.15.88:0/2915062009 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9be5d412-2421-4c86-8d81-540b24dc2616", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fde6802ca80 con 0x7fde70146310 2024-05-15T12:31:46.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.961+0000 7fde627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9be5d412-2421-4c86-8d81-540b24dc2616", "id": 4}] 2024-05-15T12:31:46.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.961+0000 7fde627fc640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:46.950 INFO:teuthology.orchestra.run.smithi088.stdout:4 2024-05-15T12:31:46.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.961+0000 7fde77bf1640 10 librados: watch_flush enter 2024-05-15T12:31:46.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.961+0000 7fde77bf1640 10 librados: watch_flush exit 2024-05-15T12:31:46.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.961+0000 7fde77bf1640 20 client.4143.objecter shutdown clearing up homeless session... 2024-05-15T12:31:46.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.961+0000 7fde77bf1640 10 client.4143.objecter successfully canceled tick 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 10 monclient: shutdown 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 1 -- 172.21.15.88:0/2915062009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde70146310 msgr2=0x7fde701d8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 1 --2- 172.21.15.88:0/2915062009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde70146310 0x7fde701d8ab0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde68036040 tx=0x7fde68004a40 comp rx=0 tx=0).stop 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 1 -- 172.21.15.88:0/2915062009 shutdown_connections 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 1 --2- 172.21.15.88:0/2915062009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde70146310 0x7fde701d8ab0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 1 --2- 172.21.15.88:0/2915062009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7013f990 0x7fde701d8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 1 --2- 172.21.15.88:0/2915062009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde700b5410 0x7fde70144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 1 -- 172.21.15.88:0/2915062009 >> 172.21.15.88:0/2915062009 conn(0x7fde70066f00 msgr2=0x7fde7013dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 1 -- 172.21.15.88:0/2915062009 shutdown_connections 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 1 -- 172.21.15.88:0/2915062009 wait complete. 2024-05-15T12:31:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:46.962+0000 7fde77bf1640 1 librados: shutdown 2024-05-15T12:31:46.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new d05e6dcc-742b-448b-853f-375b06280a8d 5 2024-05-15T12:31:47.040 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:47.050+0000 7f83ee842040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:31:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.071+0000 7f59ec982640 10 monclient: get_monmap_and_config 2024-05-15T12:31:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.071+0000 7f59ec982640 10 monclient: build_initial_monmap 2024-05-15T12:31:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.071+0000 7f59ec982640 10 monclient: monmap: 2024-05-15T12:31:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:31:47.071733+0000 2024-05-15T12:31:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:31:47.071733+0000 2024-05-15T12:31:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:47.061 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:47.061 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:47.061 INFO:teuthology.orchestra.run.smithi088.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:47.061 INFO:teuthology.orchestra.run.smithi088.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:47.061 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.071+0000 7f59ec982640 1 Processor -- start 2024-05-15T12:31:47.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 1 -- start start 2024-05-15T12:31:47.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 10 monclient: init 2024-05-15T12:31:47.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:47.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-05-15T12:31:47.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59e80b5410 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:47.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f59e81400a0 0x7f59e80a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 10 monclient(hunting): picked mon.noname-c con 0x7f59e81400a0 addr v2:172.21.15.88:3300/0 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59e8145c00 addr v1:172.21.15.88:6789/0 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e80ab470 con 0x7f59e80b5410 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e80aa0c0 con 0x7f59e8145c00 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e80b57f0 con 0x7f59e81400a0 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.072+0000 7f59ec982640 10 monclient(hunting): authenticate will time out at 1480.332609s 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.073+0000 7f59e5d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f59e81400a0 0x7f59e80a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.073+0000 7f59e659c640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f59e80b5410 0x7f59e813f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.88:41360/0 (socket says 172.21.15.88:41360) 2024-05-15T12:31:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.073+0000 7f59e659c640 1 -- 172.21.15.88:0/3722823585 learned_addr learned my addr 172.21.15.88:0/3722823585 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-05-15T12:31:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.074+0000 7f59e5d9b640 10 monclient(hunting): get_auth_request con 0x7f59e81400a0 auth_method 0 2024-05-15T12:31:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.074+0000 7f59e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.074+0000 7f59e5d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.074+0000 7f59e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.074+0000 7f59e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.074+0000 7f59e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.074+0000 7f59e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.076+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825707333 0 0) 0x7f59e80ab470 con 0x7f59e80b5410 2024-05-15T12:31:47.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.076+0000 7f59e559a640 10 monclient(hunting): _negotiate 2024-05-15T12:31:47.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.076+0000 7f59e559a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.076+0000 7f59e559a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.076+0000 7f59e559a640 10 monclient(hunting): my global_id is 4146 2024-05-15T12:31:47.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.076+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d4003520 con 0x7f59e80b5410 2024-05-15T12:31:47.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 <== mon.0 v1:172.21.15.49:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288403726 0 0) 0x7f59d4003520 con 0x7f59e80b5410 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 >> v2:172.21.15.88:3300/0 conn(0x7f59e81400a0 msgr2=0x7f59e80a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 --2- 172.21.15.88:0/3722823585 >> v2:172.21.15.88:3300/0 conn(0x7f59e81400a0 0x7f59e80a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 >> v1:172.21.15.88:6789/0 conn(0x7f59e8145c00 legacy=0x7f59e8146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient: _finish_hunting 0 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 monclient: found mon.noname-a-legacy 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.49:6789/0 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 --> v1:172.21.15.49:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e80aa5e0 con 0x7f59e80b5410 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient: _finish_auth 0 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient: _check_auth_tickets 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59ec982640 5 monclient: authenticate success, global_id 4146 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59ec982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 <== mon.0 v1:172.21.15.49:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2686122088 0 0) 0x7f59d8001670 con 0x7f59e80b5410 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient: dump: 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] but i'm connected to v1:172.21.15.49:6789/0, reconnecting 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 >> v1:172.21.15.49:6789/0 conn(0x7f59e80b5410 legacy=0x7f59e813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59d4003a20 0x7f59d4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient(hunting): picked mon.b con 0x7f59d4003a20 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59d4004800 0x7f59d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient(hunting): picked mon.a con 0x7f59d4004800 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59d4005230 0x7f59d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient(hunting): picked mon.c con 0x7f59d4005230 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80b57f0 con 0x7f59d4004800 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d400c190 con 0x7f59d4003a20 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d400c390 con 0x7f59d4005230 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59e559a640 10 monclient(hunting): _finish_auth 0 2024-05-15T12:31:47.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.079+0000 7f59ec982640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:47.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.080+0000 7f59e6d9d640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59d4005230 0x7f59d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.080+0000 7f59e6d9d640 10 monclient(hunting): get_auth_request con 0x7f59d4005230 auth_method 0 2024-05-15T12:31:47.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.080+0000 7f59e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.080+0000 7f59e6d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.080+0000 7f59e6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:47.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.081+0000 7f59e5d9b640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59d4004800 0x7f59d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.081+0000 7f59e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.081+0000 7f59e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.081+0000 7f59e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.082+0000 7f59e5d9b640 10 monclient(hunting): get_auth_request con 0x7f59d4004800 auth_method 0 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.082+0000 7f59e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.082+0000 7f59e5d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.082+0000 7f59e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.083+0000 7f59e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.083+0000 7f59e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.083+0000 7f59e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.083+0000 7f59e659c640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59d4003a20 0x7f59d4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.083+0000 7f59e659c640 10 monclient(hunting): get_auth_request con 0x7f59d4003a20 auth_method 0 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.083+0000 7f59e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.083+0000 7f59e659c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.083+0000 7f59e659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 10 monclient(hunting): handle_auth_done global_id 4146 payload 293 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 1 -- 172.21.15.88:0/3722823585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59d4003a20 msgr2=0x7f59d4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:47.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59d4003a20 0x7f59d4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 1 -- 172.21.15.88:0/3722823585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59d4004800 msgr2=0x7f59d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59d4004800 0x7f59d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 10 monclient: _finish_hunting 0 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 1 monclient: found mon.c 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 1 -- 172.21.15.88:0/3722823585 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d4003520 con 0x7f59d4005230 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 10 monclient: _finish_auth 0 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 10 monclient: _check_auth_tickets 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.084+0000 7f59e6d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.085+0000 7f59e6d9d640 1 -- 172.21.15.88:0/3722823585 reap_dead start 2024-05-15T12:31:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.087+0000 7f59e6d9d640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59d4005230 0x7f59d400b9c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f59e0001e50 tx=0x7f59d4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e894022077e2ff99 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:47.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0034020 con 0x7f59d4005230 2024-05-15T12:31:47.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59e559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59e559a640 10 monclient: dump: 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e0003b90 con 0x7f59d4005230 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59e559a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59e559a640 1 -- 172.21.15.88:0/3722823585 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0001a70 con 0x7f59d4005230 2024-05-15T12:31:47.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59ec982640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59ec982640 4 set_mon_vals no callback set 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59e559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.091+0000 7f59e559a640 10 monclient: dump: 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.082 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.092+0000 7f59ec982640 10 monclient: shutdown 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.092+0000 7f59ec982640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.092+0000 7f59ec982640 1 -- 172.21.15.88:0/3722823585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59d4005230 msgr2=0x7f59d400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.092+0000 7f59ec982640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59d4005230 0x7f59d400b9c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f59e0001e50 tx=0x7f59d4005830 comp rx=0 tx=0).stop 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.092+0000 7f59ec982640 1 -- 172.21.15.88:0/3722823585 shutdown_connections 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.092+0000 7f59ec982640 1 --2- 172.21.15.88:0/3722823585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59d4005230 0x7f59d400b9c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.092+0000 7f59ec982640 1 -- 172.21.15.88:0/3722823585 >> 172.21.15.88:0/3722823585 conn(0x7f59e8066f00 msgr2=0x7f59e8067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.092+0000 7f59ec982640 1 -- 172.21.15.88:0/3722823585 shutdown_connections 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.092+0000 7f59ec982640 1 -- 172.21.15.88:0/3722823585 wait complete. 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.093+0000 7f59ec982640 10 monclient: build_initial_monmap 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.093+0000 7f59ec982640 10 monclient: monmap: 2024-05-15T12:31:47.085 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.093+0000 7f59ec982640 1 librados: starting msgr at 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.093+0000 7f59ec982640 1 librados: starting objecter 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 1 Processor -- start 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 1 -- start start 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 1 librados: setting wanted keys 2024-05-15T12:31:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 1 librados: calling monclient init 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient: init 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59d4005230 0x7f59e81cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient(hunting): picked mon.noname-b con 0x7f59d4005230 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59e81cc2d0 0x7f59e81ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient(hunting): picked mon.noname-c con 0x7f59e81cc2d0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59e81cf090 0x7f59e81daff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient(hunting): picked mon.noname-a con 0x7f59e81cf090 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80a9960 con 0x7f59e81cf090 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80a9ea0 con 0x7f59d4005230 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80a9ae0 con 0x7f59e81cc2d0 2024-05-15T12:31:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:47.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.094+0000 7f59ec982640 10 monclient(hunting): authenticate will time out at 1480.354516s 2024-05-15T12:31:47.088 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f8a32e2d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:31:47.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59d4005230 0x7f59e81cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e5d9b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59e81cc2d0 0x7f59e81ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59e81cf090 0x7f59e81daff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59e81cf090 0x7f59e81daff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.88:57946/0 (socket says 172.21.15.88:57946) 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e6d9d640 1 -- 172.21.15.88:0/2426562925 learned_addr learned my addr 172.21.15.88:0/2426562925 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e6d9d640 10 monclient(hunting): get_auth_request con 0x7f59e81cf090 auth_method 0 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e6d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e5d9b640 10 monclient(hunting): get_auth_request con 0x7f59e81cc2d0 auth_method 0 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e5d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e659c640 10 monclient(hunting): get_auth_request con 0x7f59d4005230 auth_method 0 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e659c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.095+0000 7f59e659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.096+0000 7f59e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.096+0000 7f59e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.096+0000 7f59e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.096+0000 7f59e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.096+0000 7f59e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.096+0000 7f59e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.096+0000 7f59e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.096+0000 7f59e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.096+0000 7f59e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.099+0000 7f59e6d9d640 10 monclient(hunting): handle_auth_done global_id 4149 payload 824 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.099+0000 7f59e6d9d640 1 -- 172.21.15.88:0/2426562925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59e81cc2d0 msgr2=0x7f59e81ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.099+0000 7f59e6d9d640 1 --2- 172.21.15.88:0/2426562925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59e81cc2d0 0x7f59e81ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.099+0000 7f59e6d9d640 1 -- 172.21.15.88:0/2426562925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59d4005230 msgr2=0x7f59e81cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.099+0000 7f59e6d9d640 1 --2- 172.21.15.88:0/2426562925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59d4005230 0x7f59e81cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.099+0000 7f59e6d9d640 10 monclient: _finish_hunting 0 2024-05-15T12:31:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.099+0000 7f59e6d9d640 1 monclient: found mon.noname-a 2024-05-15T12:31:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.099+0000 7f59e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.099+0000 7f59e6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:31:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.099+0000 7f59e6d9d640 1 -- 172.21.15.88:0/2426562925 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e81cd6e0 con 0x7f59e81cf090 2024-05-15T12:31:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.101+0000 7f59e6d9d640 1 --2- 172.21.15.88:0/2426562925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59e81cf090 0x7f59e81daff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f59e0001eb0 tx=0x7f59e00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.101+0000 7f59deffd640 10 client.?.objecter ms_handle_connect 0x7f59e81cf090 2024-05-15T12:31:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.101+0000 7f59deffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:47.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 1 -- 172.21.15.88:0/2426562925 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0006d50 con 0x7f59e81cf090 2024-05-15T12:31:47.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:31:47.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 10 monclient: dump: 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 10 monclient: _finish_auth 0 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 10 monclient: _check_auth_tickets 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 1 -- 172.21.15.88:0/2426562925 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e0001250 con 0x7f59e81cf090 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 1 -- 172.21.15.88:0/2426562925 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0007270 con 0x7f59e81cf090 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59ec982640 5 monclient: authenticate success, global_id 4149 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59ec982640 10 monclient: _renew_subs 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59ec982640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59ec982640 1 -- 172.21.15.88:0/2426562925 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e81ce450 con 0x7f59e81cf090 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59deffd640 10 monclient: dump: 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.097 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59e4d99640 4 set_mon_vals no callback set 2024-05-15T12:31:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59ec982640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59ec982640 10 monclient: _renew_subs 2024-05-15T12:31:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59ec982640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:31:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59ec982640 1 -- 172.21.15.88:0/2426562925 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e80b1d60 con 0x7f59e81cf090 2024-05-15T12:31:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.105+0000 7f59ec982640 1 librados: init done 2024-05-15T12:31:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.106+0000 7f59ec982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.106+0000 7f59ec982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.106+0000 7f59ec982640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:31:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.106+0000 7f59ec982640 1 -- 172.21.15.88:0/2426562925 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e804f0e0 con 0x7f59e81cf090 2024-05-15T12:31:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.107+0000 7f59deffd640 1 -- 172.21.15.88:0/2426562925 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f59e0043020 con 0x7f59e81cf090 2024-05-15T12:31:47.193 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:47.203+0000 7f83ee842040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:31:47.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.213+0000 7f59deffd640 1 -- 172.21.15.88:0/2426562925 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f59e003a460 con 0x7f59e81cf090 2024-05-15T12:31:47.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.213+0000 7f59deffd640 10 client.4149.objecter ms_dispatch 0x7f59e8000b90 osd_map(7..7 src has 1..7) v4 2024-05-15T12:31:47.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.213+0000 7f59deffd640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-05-15T12:31:47.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.213+0000 7f59deffd640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-05-15T12:31:47.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.213+0000 7f59deffd640 20 client.4149.objecter dump_active .. 0 homeless 2024-05-15T12:31:47.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.213+0000 7f59deffd640 1 -- 172.21.15.88:0/2426562925 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f59e004a020 con 0x7f59e81cf090 2024-05-15T12:31:47.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.213+0000 7f59deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:47.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.213+0000 7f59deffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:47.225 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:47.236+0000 7f8a32e2d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:31:47.245 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:47.255+0000 7f83ee842040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:31:47.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.285+0000 7f59ec982640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d05e6dcc-742b-448b-853f-375b06280a8d", "id": 5}] 2024-05-15T12:31:47.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.285+0000 7f59ec982640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d05e6dcc-742b-448b-853f-375b06280a8d", "id": 5}] 2024-05-15T12:31:47.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.285+0000 7f59ec982640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:31:47.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.285+0000 7f59ec982640 1 -- 172.21.15.88:0/2426562925 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d05e6dcc-742b-448b-853f-375b06280a8d", "id": 5} v 0) v1 -- 0x7f59e8146ad0 con 0x7f59e81cf090 2024-05-15T12:31:47.275 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:47.286+0000 7f8a32e2d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:31:47.321 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:47.331+0000 7f83ee842040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:31:47.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.332+0000 7f59deffd640 1 -- 172.21.15.88:0/2426562925 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d05e6dcc-742b-448b-853f-375b06280a8d", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f59e00077a0 con 0x7f59e81cf090 2024-05-15T12:31:47.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.332+0000 7f59deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d05e6dcc-742b-448b-853f-375b06280a8d", "id": 5}] 2024-05-15T12:31:47.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.332+0000 7f59deffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:47.322 INFO:teuthology.orchestra.run.smithi088.stdout:5 2024-05-15T12:31:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 10 librados: watch_flush enter 2024-05-15T12:31:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 10 librados: watch_flush exit 2024-05-15T12:31:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 20 client.4149.objecter shutdown clearing up homeless session... 2024-05-15T12:31:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 10 client.4149.objecter successfully canceled tick 2024-05-15T12:31:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 10 monclient: shutdown 2024-05-15T12:31:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 1 -- 172.21.15.88:0/2426562925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59e81cf090 msgr2=0x7f59e81daff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 1 --2- 172.21.15.88:0/2426562925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59e81cf090 0x7f59e81daff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f59e0001eb0 tx=0x7f59e00014a0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 1 -- 172.21.15.88:0/2426562925 shutdown_connections 2024-05-15T12:31:47.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 1 --2- 172.21.15.88:0/2426562925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59e81cf090 0x7f59e81daff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 1 --2- 172.21.15.88:0/2426562925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59e81cc2d0 0x7f59e81ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 1 --2- 172.21.15.88:0/2426562925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59d4005230 0x7f59e81cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 1 -- 172.21.15.88:0/2426562925 >> 172.21.15.88:0/2426562925 conn(0x7f59e8066f00 msgr2=0x7f59e813d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:47.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 1 -- 172.21.15.88:0/2426562925 shutdown_connections 2024-05-15T12:31:47.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.333+0000 7f59ec982640 1 -- 172.21.15.88:0/2426562925 wait complete. 2024-05-15T12:31:47.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.334+0000 7f59ec982640 1 librados: shutdown 2024-05-15T12:31:47.334 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 341b7da5-8516-462d-b852-e867a5dd5e95 6 2024-05-15T12:31:47.348 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:47.360+0000 7f8a32e2d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:31:47.369 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:47.379+0000 7f83ee842040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:31:47.396 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:47.407+0000 7f8a32e2d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:31:47.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.447+0000 7f216acea640 10 monclient: get_monmap_and_config 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.447+0000 7f216acea640 10 monclient: build_initial_monmap 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.447+0000 7f216acea640 10 monclient: monmap: 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:31:47.447441+0000 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:31:47.447441+0000 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:47.436 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.447+0000 7f216acea640 1 Processor -- start 2024-05-15T12:31:47.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.447+0000 7f216acea640 1 -- start start 2024-05-15T12:31:47.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.447+0000 7f216acea640 10 monclient: init 2024-05-15T12:31:47.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:47.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-05-15T12:31:47.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21640b5410 addr v1:172.21.15.88:6789/0 2024-05-15T12:31:47.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f21641400a0 0x7f21640a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 10 monclient(hunting): picked mon.noname-a con 0x7f21641400a0 addr v2:172.21.15.49:3300/0 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2164145c00 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21640ab470 con 0x7f2164145c00 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21640aa0c0 con 0x7f21640b5410 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21640b57f0 con 0x7f21641400a0 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.448+0000 7f216acea640 10 monclient(hunting): authenticate will time out at 1480.708400s 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.449+0000 7f2163fff640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f21641400a0 0x7f21640a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.449+0000 7f2169287640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f2164145c00 0x7f2164146010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.88:41374/0 (socket says 172.21.15.88:41374) 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.449+0000 7f2169287640 1 -- 172.21.15.88:0/3852287548 learned_addr learned my addr 172.21.15.88:0/3852287548 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.449+0000 7f2163fff640 10 monclient(hunting): get_auth_request con 0x7f21641400a0 auth_method 0 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.449+0000 7f2163fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.449+0000 7f2163fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.449+0000 7f2163fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.450+0000 7f2163fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.450+0000 7f2163fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.450+0000 7f2163fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.452+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094833726 0 0) 0x7f21640aa0c0 con 0x7f21640b5410 2024-05-15T12:31:47.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.452+0000 7f21637fe640 10 monclient(hunting): _negotiate 2024-05-15T12:31:47.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.452+0000 7f21637fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.452+0000 7f21637fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.452+0000 7f21637fe640 10 monclient(hunting): my global_id is 4144 2024-05-15T12:31:47.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.452+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2150003520 con 0x7f21640b5410 2024-05-15T12:31:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3136120331 0 0) 0x7f2150003520 con 0x7f21640b5410 2024-05-15T12:31:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 >> v2:172.21.15.49:3300/0 conn(0x7f21641400a0 msgr2=0x7f21640a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 --2- 172.21.15.88:0/3852287548 >> v2:172.21.15.49:3300/0 conn(0x7f21641400a0 0x7f21640a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 >> v1:172.21.15.49:6789/0 conn(0x7f2164145c00 legacy=0x7f2164146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient: _finish_hunting 0 2024-05-15T12:31:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 monclient: found mon.noname-c-legacy 2024-05-15T12:31:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.88:6789/0 2024-05-15T12:31:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21640aa5e0 con 0x7f21640b5410 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient: _finish_auth 0 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient: _check_auth_tickets 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2686122088 0 0) 0x7f215c0019f0 con 0x7f21640b5410 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f216acea640 5 monclient: authenticate success, global_id 4144 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f216acea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient: dump: 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] but i'm connected to v1:172.21.15.88:6789/0, reconnecting 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 >> v1:172.21.15.88:6789/0 conn(0x7f21640b5410 legacy=0x7f216413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 --2- 172.21.15.88:0/3852287548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2150003a70 0x7f2150003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient(hunting): picked mon.a con 0x7f2150003a70 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 --2- 172.21.15.88:0/3852287548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2150004800 0x7f2150004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient(hunting): picked mon.c con 0x7f2150004800 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 --2- 172.21.15.88:0/3852287548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21500052e0 0x7f215000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient(hunting): picked mon.b con 0x7f21500052e0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21640b57f0 con 0x7f2150003a70 2024-05-15T12:31:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215000c1b0 con 0x7f21500052e0 2024-05-15T12:31:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215000c3b0 con 0x7f2150004800 2024-05-15T12:31:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f21637fe640 10 monclient(hunting): _finish_auth 0 2024-05-15T12:31:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.456+0000 7f216acea640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:47.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.459+0000 7f2169287640 1 --2- 172.21.15.88:0/3852287548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21500052e0 0x7f215000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.459+0000 7f2163fff640 1 --2- 172.21.15.88:0/3852287548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2150004800 0x7f2150004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2163fff640 10 monclient(hunting): get_auth_request con 0x7f2150004800 auth_method 0 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2163fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2163fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2163fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2169287640 10 monclient(hunting): get_auth_request con 0x7f21500052e0 auth_method 0 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2169287640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2169287640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2169287640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2169287640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2169287640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2169287640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.460+0000 7f2163fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.461+0000 7f2163fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.461+0000 7f2163fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:31:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 10 monclient(hunting): handle_auth_done global_id 4144 payload 293 2024-05-15T12:31:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 1 -- 172.21.15.88:0/3852287548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2150004800 msgr2=0x7f2150004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 1 --2- 172.21.15.88:0/3852287548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2150004800 0x7f2150004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 1 -- 172.21.15.88:0/3852287548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2150003a70 msgr2=0x7f2150003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 1 --2- 172.21.15.88:0/3852287548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2150003a70 0x7f2150003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 10 monclient: _finish_hunting 0 2024-05-15T12:31:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 1 monclient: found mon.b 2024-05-15T12:31:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 1 -- 172.21.15.88:0/3852287548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150003520 con 0x7f21500052e0 2024-05-15T12:31:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 10 monclient: _finish_auth 0 2024-05-15T12:31:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 10 monclient: _check_auth_tickets 2024-05-15T12:31:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.462+0000 7f2169287640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.463+0000 7f2169287640 1 -- 172.21.15.88:0/3852287548 reap_dead start 2024-05-15T12:31:47.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.464+0000 7f2169287640 1 --2- 172.21.15.88:0/3852287548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21500052e0 0x7f215000b9c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2154001810 tx=0x7f2150005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7635a6e81fecb88f server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:47.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154001460 con 0x7f21500052e0 2024-05-15T12:31:47.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f21637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f21637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:47.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f21637fe640 10 monclient: dump: 2024-05-15T12:31:47.460 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.460 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.460 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2154007070 con 0x7f21500052e0 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f216acea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f21637fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f216acea640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f21637fe640 1 -- 172.21.15.88:0/3852287548 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154007440 con 0x7f21500052e0 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f21637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f216acea640 4 set_mon_vals no callback set 2024-05-15T12:31:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f21637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f21637fe640 10 monclient: dump: 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.462 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f216acea640 10 monclient: shutdown 2024-05-15T12:31:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.471+0000 7f216acea640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.472+0000 7f216acea640 1 -- 172.21.15.88:0/3852287548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21500052e0 msgr2=0x7f215000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.472+0000 7f216acea640 1 --2- 172.21.15.88:0/3852287548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21500052e0 0x7f215000b9c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2154001810 tx=0x7f2150005830 comp rx=0 tx=0).stop 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.472+0000 7f216acea640 1 -- 172.21.15.88:0/3852287548 shutdown_connections 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.472+0000 7f216acea640 1 --2- 172.21.15.88:0/3852287548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21500052e0 0x7f215000b9c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.472+0000 7f216acea640 1 -- 172.21.15.88:0/3852287548 >> 172.21.15.88:0/3852287548 conn(0x7f2164066f00 msgr2=0x7f2164067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.472+0000 7f216acea640 1 -- 172.21.15.88:0/3852287548 shutdown_connections 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.472+0000 7f216acea640 1 -- 172.21.15.88:0/3852287548 wait complete. 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.472+0000 7f216acea640 10 monclient: build_initial_monmap 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.472+0000 7f216acea640 10 monclient: monmap: 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 librados: starting msgr at 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 librados: starting objecter 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 Processor -- start 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 -- start start 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 librados: setting wanted keys 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 librados: calling monclient init 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient: init 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21500052e0 0x7f2164143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient(hunting): picked mon.noname-c con 0x7f21500052e0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2164143ec0 0x7f21641db070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient(hunting): picked mon.noname-b con 0x7f2164143ec0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21641db5b0 0x7f21641d7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient(hunting): picked mon.noname-a con 0x7f21641db5b0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21640a9960 con 0x7f21641db5b0 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21640a9ea0 con 0x7f2164143ec0 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21640a9ae0 con 0x7f21500052e0 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.473+0000 7f216acea640 10 monclient(hunting): authenticate will time out at 1480.733712s 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.474+0000 7f2163fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2164143ec0 0x7f21641db070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.474+0000 7f2168a86640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21500052e0 0x7f2164143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.474+0000 7f2163fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2164143ec0 0x7f21641db070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38944/0 (socket says 172.21.15.88:38944) 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.474+0000 7f2163fff640 1 -- 172.21.15.88:0/1776378423 learned_addr learned my addr 172.21.15.88:0/1776378423 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.474+0000 7f2163fff640 10 monclient(hunting): get_auth_request con 0x7f2164143ec0 auth_method 0 2024-05-15T12:31:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.474+0000 7f2163fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.474+0000 7f2163fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.474+0000 7f2163fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.475+0000 7f2168a86640 10 monclient(hunting): get_auth_request con 0x7f21500052e0 auth_method 0 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.475+0000 7f2168a86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.475+0000 7f2168a86640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.475+0000 7f2168a86640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.475+0000 7f2163fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.475+0000 7f2163fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.475+0000 7f2163fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.475+0000 7f2168a86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.475+0000 7f2168a86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.475+0000 7f2168a86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.478+0000 7f2163fff640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2024-05-15T12:31:47.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.478+0000 7f2163fff640 1 -- 172.21.15.88:0/1776378423 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21500052e0 msgr2=0x7f2164143790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.478+0000 7f2163fff640 1 --2- 172.21.15.88:0/1776378423 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21500052e0 0x7f2164143790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.478+0000 7f2163fff640 1 -- 172.21.15.88:0/1776378423 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21641db5b0 msgr2=0x7f21641d7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.478+0000 7f2163fff640 1 --2- 172.21.15.88:0/1776378423 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21641db5b0 0x7f21641d7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.478+0000 7f2163fff640 10 monclient: _finish_hunting 0 2024-05-15T12:31:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.478+0000 7f2163fff640 1 monclient: found mon.noname-b 2024-05-15T12:31:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.478+0000 7f2163fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.478+0000 7f2163fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.478+0000 7f2163fff640 1 -- 172.21.15.88:0/1776378423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21641455c0 con 0x7f2164143ec0 2024-05-15T12:31:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.480+0000 7f2163fff640 1 --2- 172.21.15.88:0/1776378423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2164143ec0 0x7f21641db070 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f214c011960 tx=0x7f214c011e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.480+0000 7f21617fa640 10 client.?.objecter ms_handle_connect 0x7f2164143ec0 2024-05-15T12:31:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.480+0000 7f21617fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:47.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 1 -- 172.21.15.88:0/1776378423 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214c019070 con 0x7f2164143ec0 2024-05-15T12:31:47.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 10 monclient: dump: 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 10 monclient: _finish_auth 0 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 10 monclient: _check_auth_tickets 2024-05-15T12:31:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 1 -- 172.21.15.88:0/1776378423 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214c004240 con 0x7f2164143ec0 2024-05-15T12:31:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 1 -- 172.21.15.88:0/1776378423 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214c00f820 con 0x7f2164143ec0 2024-05-15T12:31:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f21617fa640 10 monclient: dump: 2024-05-15T12:31:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f216acea640 5 monclient: authenticate success, global_id 4147 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f216acea640 10 monclient: _renew_subs 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f216acea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f216acea640 1 -- 172.21.15.88:0/1776378423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21641446b0 con 0x7f2164143ec0 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f2162ffd640 4 set_mon_vals no callback set 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f216acea640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f216acea640 10 monclient: _renew_subs 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f216acea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f216acea640 1 -- 172.21.15.88:0/1776378423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21641dba40 con 0x7f2164143ec0 2024-05-15T12:31:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.483+0000 7f216acea640 1 librados: init done 2024-05-15T12:31:47.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.484+0000 7f216acea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:47.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.484+0000 7f216acea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:47.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.484+0000 7f216acea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.484+0000 7f216acea640 1 -- 172.21.15.88:0/1776378423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21641444e0 con 0x7f2164143ec0 2024-05-15T12:31:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.486+0000 7f21617fa640 1 -- 172.21.15.88:0/1776378423 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f214c01c070 con 0x7f2164143ec0 2024-05-15T12:31:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.486+0000 7f21617fa640 1 -- 172.21.15.88:0/1776378423 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f214c003c00 con 0x7f2164143ec0 2024-05-15T12:31:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.486+0000 7f21617fa640 10 client.4147.objecter ms_dispatch 0x7f2164000b90 osd_map(8..8 src has 1..8) v4 2024-05-15T12:31:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.486+0000 7f21617fa640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-05-15T12:31:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.486+0000 7f21617fa640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-05-15T12:31:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.486+0000 7f21617fa640 20 client.4147.objecter dump_active .. 0 homeless 2024-05-15T12:31:47.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.606+0000 7f21617fa640 1 -- 172.21.15.88:0/1776378423 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f214c003e00 con 0x7f2164143ec0 2024-05-15T12:31:47.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.606+0000 7f21617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:47.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.606+0000 7f21617fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:47.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.676+0000 7f216acea640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "341b7da5-8516-462d-b852-e867a5dd5e95", "id": 6}] 2024-05-15T12:31:47.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.676+0000 7f216acea640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "341b7da5-8516-462d-b852-e867a5dd5e95", "id": 6}] 2024-05-15T12:31:47.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.676+0000 7f216acea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.676+0000 7f216acea640 1 -- 172.21.15.88:0/1776378423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "341b7da5-8516-462d-b852-e867a5dd5e95", "id": 6} v 0) v1 -- 0x7f216404f0e0 con 0x7f2164143ec0 2024-05-15T12:31:47.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.753+0000 7f21617fa640 1 -- 172.21.15.88:0/1776378423 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "341b7da5-8516-462d-b852-e867a5dd5e95", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f214c0037a0 con 0x7f2164143ec0 2024-05-15T12:31:47.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.753+0000 7f21617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "341b7da5-8516-462d-b852-e867a5dd5e95", "id": 6}] 2024-05-15T12:31:47.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.753+0000 7f21617fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:47.743 INFO:teuthology.orchestra.run.smithi088.stdout:6 2024-05-15T12:31:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.754+0000 7f216acea640 10 librados: watch_flush enter 2024-05-15T12:31:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.754+0000 7f216acea640 10 librados: watch_flush exit 2024-05-15T12:31:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.754+0000 7f216acea640 20 client.4147.objecter shutdown clearing up homeless session... 2024-05-15T12:31:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.754+0000 7f216acea640 10 client.4147.objecter successfully canceled tick 2024-05-15T12:31:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.754+0000 7f216acea640 10 monclient: shutdown 2024-05-15T12:31:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.754+0000 7f216acea640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:47.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.754+0000 7f216acea640 1 -- 172.21.15.88:0/1776378423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2164143ec0 msgr2=0x7f21641db070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.754+0000 7f216acea640 1 --2- 172.21.15.88:0/1776378423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2164143ec0 0x7f21641db070 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f214c011960 tx=0x7f214c011e20 comp rx=0 tx=0).stop 2024-05-15T12:31:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.755+0000 7f216acea640 1 -- 172.21.15.88:0/1776378423 shutdown_connections 2024-05-15T12:31:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.755+0000 7f216acea640 1 --2- 172.21.15.88:0/1776378423 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21641db5b0 0x7f21641d7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.755+0000 7f216acea640 1 --2- 172.21.15.88:0/1776378423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2164143ec0 0x7f21641db070 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.755+0000 7f216acea640 1 --2- 172.21.15.88:0/1776378423 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21500052e0 0x7f2164143790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.755+0000 7f216acea640 1 -- 172.21.15.88:0/1776378423 >> 172.21.15.88:0/1776378423 conn(0x7f2164066f00 msgr2=0x7f216413de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.755+0000 7f216acea640 1 -- 172.21.15.88:0/1776378423 shutdown_connections 2024-05-15T12:31:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.755+0000 7f216acea640 1 -- 172.21.15.88:0/1776378423 wait complete. 2024-05-15T12:31:47.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.755+0000 7f216acea640 1 librados: shutdown 2024-05-15T12:31:47.754 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new ca94bfcf-3295-4f81-a69e-ba59dd3720c5 7 2024-05-15T12:31:47.880 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:47.890+0000 7f83ee842040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:31:47.892 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:47.903+0000 7f8a32e2d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:31:47.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.905+0000 7f8331344640 10 monclient: get_monmap_and_config 2024-05-15T12:31:47.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.905+0000 7f8331344640 10 monclient: build_initial_monmap 2024-05-15T12:31:47.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.905+0000 7f8331344640 10 monclient: monmap: 2024-05-15T12:31:47.894 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:47.894 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.894 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:31:47.905794+0000 2024-05-15T12:31:47.894 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:31:47.905794+0000 2024-05-15T12:31:47.894 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:47.894 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.894 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:47.895 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:47.895 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:47.895 INFO:teuthology.orchestra.run.smithi088.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:47.895 INFO:teuthology.orchestra.run.smithi088.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:47.895 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 1 Processor -- start 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 1 -- start start 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 10 monclient: init 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f832c0b53e0 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f832c142870 0x7f832c142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 10 monclient(hunting): picked mon.noname-a con 0x7f832c142870 addr v2:172.21.15.49:3300/0 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f832c1431f0 0x7f832c13f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 10 monclient(hunting): picked mon.noname-c con 0x7f832c1431f0 addr v2:172.21.15.88:3300/0 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c0ab230 con 0x7f832c0b53e0 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832c0b57c0 con 0x7f832c142870 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832c0aa230 con 0x7f832c1431f0 2024-05-15T12:31:47.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.906+0000 7f8331344640 10 monclient(hunting): authenticate will time out at 1481.166462s 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f832c1431f0 0x7f832c13f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832a7fc640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f832c142870 0x7f832c142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f832c1431f0 0x7f832c13f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38952/0 (socket says 172.21.15.88:38952) 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832b7fe640 1 -- 172.21.15.88:0/3876733741 learned_addr learned my addr 172.21.15.88:0/3876733741 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832b7fe640 10 monclient(hunting): get_auth_request con 0x7f832c1431f0 auth_method 0 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832a7fc640 10 monclient(hunting): get_auth_request con 0x7f832c142870 auth_method 0 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.907+0000 7f832a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.908+0000 7f832b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.908+0000 7f832b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.908+0000 7f832b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.908+0000 7f832a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.908+0000 7f832a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.908+0000 7f832a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.909+0000 7f832b7fe640 10 monclient(hunting): handle_auth_done global_id 4150 payload 274 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.909+0000 7f832b7fe640 1 -- 172.21.15.88:0/3876733741 >> v2:172.21.15.49:3300/0 conn(0x7f832c142870 msgr2=0x7f832c142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.909+0000 7f832b7fe640 1 --2- 172.21.15.88:0/3876733741 >> v2:172.21.15.49:3300/0 conn(0x7f832c142870 0x7f832c142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.909+0000 7f832b7fe640 1 -- 172.21.15.88:0/3876733741 >> v1:172.21.15.49:6789/0 conn(0x7f832c0b53e0 legacy=0x7f832c0a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.909+0000 7f832b7fe640 10 monclient: _finish_hunting 0 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.909+0000 7f832b7fe640 1 monclient: found mon.noname-c 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.909+0000 7f832b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.909+0000 7f832b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.909+0000 7f832b7fe640 1 -- 172.21.15.88:0/3876733741 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c0ab510 con 0x7f832c1431f0 2024-05-15T12:31:47.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.911+0000 7f832b7fe640 1 --2- 172.21.15.88:0/3876733741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 0x7f832c13f660 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f832c0b46d0 tx=0x7f832402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7386a2794a4532b server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:47.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 1 -- 172.21.15.88:0/3876733741 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832402ca80 con 0x7f832c1431f0 2024-05-15T12:31:47.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 10 monclient: dump: 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 10 monclient: _finish_auth 0 2024-05-15T12:31:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 1 -- 172.21.15.88:0/3876733741 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832402cc20 con 0x7f832c1431f0 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 1 -- 172.21.15.88:0/3876733741 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83240343f0 con 0x7f832c1431f0 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8331344640 5 monclient: authenticate success, global_id 4150 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8331344640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8331344640 4 set_mon_vals no callback set 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.914+0000 7f8329ffb640 10 monclient: dump: 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.905 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.906 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.906 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.906 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.906 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.906 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.915+0000 7f8331344640 10 monclient: shutdown 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.915+0000 7f8331344640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.915+0000 7f8331344640 1 -- 172.21.15.88:0/3876733741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 msgr2=0x7f832c13f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.915+0000 7f8331344640 1 --2- 172.21.15.88:0/3876733741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 0x7f832c13f660 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f832c0b46d0 tx=0x7f832402bee0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.915+0000 7f8331344640 1 -- 172.21.15.88:0/3876733741 shutdown_connections 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.915+0000 7f8331344640 1 --2- 172.21.15.88:0/3876733741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 0x7f832c13f660 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.915+0000 7f8331344640 1 --2- 172.21.15.88:0/3876733741 >> v2:172.21.15.49:3300/0 conn(0x7f832c142870 0x7f832c142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.915+0000 7f8331344640 1 -- 172.21.15.88:0/3876733741 >> 172.21.15.88:0/3876733741 conn(0x7f832c066ed0 msgr2=0x7f832c0672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.915+0000 7f8331344640 1 -- 172.21.15.88:0/3876733741 shutdown_connections 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.915+0000 7f8331344640 1 -- 172.21.15.88:0/3876733741 wait complete. 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.916+0000 7f8331344640 10 monclient: build_initial_monmap 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.916+0000 7f8331344640 10 monclient: monmap: 2024-05-15T12:31:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.916+0000 7f8331344640 1 librados: starting msgr at 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.916+0000 7f8331344640 1 librados: starting objecter 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.916+0000 7f8331344640 1 Processor -- start 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 1 -- start start 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 1 librados: setting wanted keys 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 1 librados: calling monclient init 2024-05-15T12:31:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient: init 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c0b53e0 0x7f832c139630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient(hunting): picked mon.noname-a con 0x7f832c0b53e0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c142870 0x7f832c1db370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient(hunting): picked mon.noname-c con 0x7f832c142870 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 0x7f832c0af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient(hunting): picked mon.noname-b con 0x7f832c1431f0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c13ff30 con 0x7f832c0b53e0 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c13fdb0 con 0x7f832c1431f0 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c1400b0 con 0x7f832c142870 2024-05-15T12:31:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.917+0000 7f8331344640 10 monclient(hunting): authenticate will time out at 1481.177399s 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 0x7f832c0af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c0b53e0 0x7f832c139630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 0x7f832c0af7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38958/0 (socket says 172.21.15.88:38958) 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832b7fe640 1 -- 172.21.15.88:0/736536082 learned_addr learned my addr 172.21.15.88:0/736536082 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832b7fe640 10 monclient(hunting): get_auth_request con 0x7f832c1431f0 auth_method 0 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832a7fc640 1 --2- 172.21.15.88:0/736536082 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c142870 0x7f832c1db370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832affd640 10 monclient(hunting): get_auth_request con 0x7f832c0b53e0 auth_method 0 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832affd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832a7fc640 10 monclient(hunting): get_auth_request con 0x7f832c142870 auth_method 0 2024-05-15T12:31:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.918+0000 7f832affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.919+0000 7f832a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.919+0000 7f832a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:47.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.919+0000 7f832a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:47.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.921+0000 7f832b7fe640 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2024-05-15T12:31:47.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.921+0000 7f832b7fe640 1 -- 172.21.15.88:0/736536082 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c142870 msgr2=0x7f832c1db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.921+0000 7f832b7fe640 1 --2- 172.21.15.88:0/736536082 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c142870 0x7f832c1db370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.921+0000 7f832b7fe640 1 -- 172.21.15.88:0/736536082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c0b53e0 msgr2=0x7f832c139630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:47.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.921+0000 7f832b7fe640 1 --2- 172.21.15.88:0/736536082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c0b53e0 0x7f832c139630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:47.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.921+0000 7f832b7fe640 10 monclient: _finish_hunting 0 2024-05-15T12:31:47.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.921+0000 7f832b7fe640 1 monclient: found mon.noname-b 2024-05-15T12:31:47.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.921+0000 7f832b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:47.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.921+0000 7f832b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.921+0000 7f832b7fe640 1 -- 172.21.15.88:0/736536082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c11ff30 con 0x7f832c1431f0 2024-05-15T12:31:47.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.923+0000 7f832b7fe640 1 --2- 172.21.15.88:0/736536082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 0x7f832c0af7f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8324009f40 tx=0x7f8324004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:47.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.923+0000 7f83137fe640 10 client.?.objecter ms_handle_connect 0x7f832c1431f0 2024-05-15T12:31:47.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.923+0000 7f83137fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:47.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 1 -- 172.21.15.88:0/736536082 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324043020 con 0x7f832c1431f0 2024-05-15T12:31:47.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 10 monclient: dump: 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 10 monclient: _finish_auth 0 2024-05-15T12:31:47.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 10 monclient: _check_auth_tickets 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 1 -- 172.21.15.88:0/736536082 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832403e450 con 0x7f832c1431f0 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f8331344640 5 monclient: authenticate success, global_id 4153 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 1 -- 172.21.15.88:0/736536082 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324009510 con 0x7f832c1431f0 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f8331344640 10 monclient: _renew_subs 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f8331344640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f8331344640 1 -- 172.21.15.88:0/736536082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c1dc000 con 0x7f832c1431f0 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83137fe640 10 monclient: dump: 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:47.919 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f83297fa640 4 set_mon_vals no callback set 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f8331344640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f8331344640 10 monclient: _renew_subs 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f8331344640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f8331344640 1 -- 172.21.15.88:0/736536082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c0afd30 con 0x7f832c1431f0 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.927+0000 7f8331344640 1 librados: init done 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.928+0000 7f8331344640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.928+0000 7f8331344640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.928+0000 7f8331344640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:47.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.928+0000 7f8331344640 1 -- 172.21.15.88:0/736536082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f832c1dbce0 con 0x7f832c1431f0 2024-05-15T12:31:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.931+0000 7f83137fe640 1 -- 172.21.15.88:0/736536082 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8324046020 con 0x7f832c1431f0 2024-05-15T12:31:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.931+0000 7f83137fe640 1 -- 172.21.15.88:0/736536082 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f8324051a90 con 0x7f832c1431f0 2024-05-15T12:31:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.931+0000 7f83137fe640 10 client.4153.objecter ms_dispatch 0x7f832c000b90 osd_map(9..9 src has 1..9) v4 2024-05-15T12:31:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.931+0000 7f83137fe640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-05-15T12:31:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.931+0000 7f83137fe640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-05-15T12:31:47.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:47.931+0000 7f83137fe640 20 client.4153.objecter dump_active .. 0 homeless 2024-05-15T12:31:48.004 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:48.014+0000 7f83ee842040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:31:48.012 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-15T12:31:48.023+0000 7f8a32e2d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:31:48.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.027+0000 7f83137fe640 1 -- 172.21.15.88:0/736536082 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f832c1dbce0 con 0x7f832c1431f0 2024-05-15T12:31:48.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.027+0000 7f83137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:48.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.027+0000 7f83137fe640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:48.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.097+0000 7f8331344640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ca94bfcf-3295-4f81-a69e-ba59dd3720c5", "id": 7}] 2024-05-15T12:31:48.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.097+0000 7f8331344640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ca94bfcf-3295-4f81-a69e-ba59dd3720c5", "id": 7}] 2024-05-15T12:31:48.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.097+0000 7f8331344640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:48.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.097+0000 7f8331344640 1 -- 172.21.15.88:0/736536082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ca94bfcf-3295-4f81-a69e-ba59dd3720c5", "id": 7} v 0) v1 -- 0x7f832c04f0e0 con 0x7f832c1431f0 2024-05-15T12:31:48.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.363+0000 7f83137fe640 1 -- 172.21.15.88:0/736536082 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 2) v1 ==== 44914+0+0 (secure 0 0 0) 0x7f8324005d10 con 0x7f832c1431f0 2024-05-15T12:31:48.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.556+0000 7f83137fe640 1 -- 172.21.15.88:0/736536082 <== mon.1 v2:172.21.15.88:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ca94bfcf-3295-4f81-a69e-ba59dd3720c5", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f832404d030 con 0x7f832c1431f0 2024-05-15T12:31:48.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.556+0000 7f83137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ca94bfcf-3295-4f81-a69e-ba59dd3720c5", "id": 7}] 2024-05-15T12:31:48.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.556+0000 7f83137fe640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:48.545 INFO:teuthology.orchestra.run.smithi088.stdout:7 2024-05-15T12:31:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 10 librados: watch_flush enter 2024-05-15T12:31:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 10 librados: watch_flush exit 2024-05-15T12:31:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 20 client.4153.objecter shutdown clearing up homeless session... 2024-05-15T12:31:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 10 client.4153.objecter successfully canceled tick 2024-05-15T12:31:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 10 monclient: shutdown 2024-05-15T12:31:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 1 -- 172.21.15.88:0/736536082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 msgr2=0x7f832c0af7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 1 --2- 172.21.15.88:0/736536082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 0x7f832c0af7f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8324009f40 tx=0x7f8324004a40 comp rx=0 tx=0).stop 2024-05-15T12:31:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 1 -- 172.21.15.88:0/736536082 shutdown_connections 2024-05-15T12:31:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 1 --2- 172.21.15.88:0/736536082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f832c1431f0 0x7f832c0af7f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 1 --2- 172.21.15.88:0/736536082 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c142870 0x7f832c1db370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 1 --2- 172.21.15.88:0/736536082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c0b53e0 0x7f832c139630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 1 -- 172.21.15.88:0/736536082 >> 172.21.15.88:0/736536082 conn(0x7f832c066ed0 msgr2=0x7f832c13c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 1 -- 172.21.15.88:0/736536082 shutdown_connections 2024-05-15T12:31:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 1 -- 172.21.15.88:0/736536082 wait complete. 2024-05-15T12:31:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-15T12:31:48.557+0000 7f8331344640 1 librados: shutdown 2024-05-15T12:31:48.556 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:31:48.557 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-15T12:31:48.557 DEBUG:teuthology.orchestra.run.smithi049:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-05-15T12:31:48.559 INFO:tasks.ceph.osd.0:Started 2024-05-15T12:31:48.559 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:31:48.559 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-15T12:31:48.560 DEBUG:teuthology.orchestra.run.smithi049:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-05-15T12:31:48.562 INFO:tasks.ceph.osd.1:Started 2024-05-15T12:31:48.562 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:31:48.562 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-15T12:31:48.562 DEBUG:teuthology.orchestra.run.smithi049:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-05-15T12:31:48.564 INFO:tasks.ceph.osd.2:Started 2024-05-15T12:31:48.564 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:31:48.564 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-15T12:31:48.565 DEBUG:teuthology.orchestra.run.smithi049:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-05-15T12:31:48.567 INFO:tasks.ceph.osd.3:Started 2024-05-15T12:31:48.567 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:31:48.567 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-15T12:31:48.567 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-05-15T12:31:48.569 INFO:tasks.ceph.osd.4:Started 2024-05-15T12:31:48.570 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:31:48.570 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-15T12:31:48.570 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-05-15T12:31:48.572 INFO:tasks.ceph.osd.5:Started 2024-05-15T12:31:48.572 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:31:48.572 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-15T12:31:48.572 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-05-15T12:31:48.574 INFO:tasks.ceph.osd.6:Started 2024-05-15T12:31:48.574 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-15T12:31:48.574 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-15T12:31:48.574 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-05-15T12:31:48.576 INFO:tasks.ceph.osd.7:Started 2024-05-15T12:31:48.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-15T12:31:48.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.721+0000 7fa5751fb640 10 monclient: get_monmap_and_config 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.721+0000 7fa5751fb640 10 monclient: build_initial_monmap 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.721+0000 7fa5751fb640 10 monclient: monmap: 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:31:48.723059+0000 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:31:48.723059+0000 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:48.712 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:48.713 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.722+0000 7fa5751fb640 1 Processor -- start 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.722+0000 7fa5751fb640 1 -- start start 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.722+0000 7fa5751fb640 10 monclient: init 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5700b5410 addr v1:172.21.15.88:6789/0 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa5701400a0 0x7fa5700a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 10 monclient(hunting): picked mon.noname-a con 0x7fa5701400a0 addr v2:172.21.15.49:3300/0 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa570145c00 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700ab470 con 0x7fa570145c00 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700aa0c0 con 0x7fa5700b5410 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:48.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5700b57f0 con 0x7fa5701400a0 2024-05-15T12:31:48.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:48.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.723+0000 7fa5751fb640 10 monclient(hunting): authenticate will time out at 1466.885970s 2024-05-15T12:31:48.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.724+0000 7fa56ed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa5700b5410 0x7fa57013f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:45440/0 (socket says 172.21.15.49:45440) 2024-05-15T12:31:48.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.724+0000 7fa56ed9d640 1 -- 172.21.15.49:0/968143016 learned_addr learned my addr 172.21.15.49:0/968143016 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:31:48.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.752+0000 7fa567fff640 1 --2- 172.21.15.49:0/968143016 >> v2:172.21.15.49:3300/0 conn(0x7fa5701400a0 0x7fa5700a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:48.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.764+0000 7fa567fff640 10 monclient(hunting): get_auth_request con 0x7fa5701400a0 auth_method 0 2024-05-15T12:31:48.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.764+0000 7fa567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:48.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.764+0000 7fa567fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:48.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.764+0000 7fa567fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:48.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.769+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:48.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.769+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:48.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.769+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:48.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.772+0000 7fa567fff640 10 monclient(hunting): handle_auth_done global_id 4164 payload 274 2024-05-15T12:31:48.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.772+0000 7fa567fff640 1 -- 172.21.15.49:0/968143016 >> v1:172.21.15.88:6789/0 conn(0x7fa5700b5410 legacy=0x7fa57013f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:48.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.772+0000 7fa567fff640 1 -- 172.21.15.49:0/968143016 >> v1:172.21.15.49:6789/0 conn(0x7fa570145c00 legacy=0x7fa570146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:48.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.772+0000 7fa567fff640 10 monclient: _finish_hunting 0 2024-05-15T12:31:48.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.772+0000 7fa567fff640 1 monclient: found mon.noname-a 2024-05-15T12:31:48.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.772+0000 7fa567fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:48.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.772+0000 7fa567fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:31:48.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.772+0000 7fa567fff640 1 -- 172.21.15.49:0/968143016 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5700aa5e0 con 0x7fa5701400a0 2024-05-15T12:31:48.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.783+0000 7fa567fff640 1 --2- 172.21.15.49:0/968143016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5701400a0 0x7fa5700a9960 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa5580097a0 tx=0x7fa55802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78e8cff015f0ec7 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:48.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 1 -- 172.21.15.49:0/968143016 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558035020 con 0x7fa5701400a0 2024-05-15T12:31:48.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:48.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:31:48.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 10 monclient: dump: 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 10 monclient: _finish_auth 0 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 10 monclient: _check_auth_tickets 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:48.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 1 -- 172.21.15.49:0/968143016 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa558031830 con 0x7fa5701400a0 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 1 -- 172.21.15.49:0/968143016 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558031c00 con 0x7fa5701400a0 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa56e59c640 10 monclient: dump: 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:48.925 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:48.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa5751fb640 5 monclient: authenticate success, global_id 4164 2024-05-15T12:31:48.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa5751fb640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:48.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.933+0000 7fa5751fb640 4 set_mon_vals no callback set 2024-05-15T12:31:48.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.934+0000 7fa5751fb640 10 monclient: shutdown 2024-05-15T12:31:48.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.934+0000 7fa5751fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:48.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.934+0000 7fa5751fb640 1 -- 172.21.15.49:0/968143016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5701400a0 msgr2=0x7fa5700a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.934+0000 7fa5751fb640 1 --2- 172.21.15.49:0/968143016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5701400a0 0x7fa5700a9960 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa5580097a0 tx=0x7fa55802b8d0 comp rx=0 tx=0).stop 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.934+0000 7fa5751fb640 1 -- 172.21.15.49:0/968143016 shutdown_connections 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.934+0000 7fa5751fb640 1 --2- 172.21.15.49:0/968143016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5701400a0 0x7fa5700a9960 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.934+0000 7fa5751fb640 1 -- 172.21.15.49:0/968143016 >> 172.21.15.49:0/968143016 conn(0x7fa570066f00 msgr2=0x7fa570067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.934+0000 7fa5751fb640 1 -- 172.21.15.49:0/968143016 shutdown_connections 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.934+0000 7fa5751fb640 1 -- 172.21.15.49:0/968143016 wait complete. 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.935+0000 7fa5751fb640 10 monclient: build_initial_monmap 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.935+0000 7fa5751fb640 10 monclient: monmap: 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:48.928 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.936+0000 7fa5751fb640 1 librados: starting msgr at 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.936+0000 7fa5751fb640 1 librados: starting objecter 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.936+0000 7fa5751fb640 1 Processor -- start 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.936+0000 7fa5751fb640 1 -- start start 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.936+0000 7fa5751fb640 1 librados: setting wanted keys 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.936+0000 7fa5751fb640 1 librados: calling monclient init 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.936+0000 7fa5751fb640 10 monclient: init 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.937+0000 7fa5751fb640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.937+0000 7fa5751fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:31:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.937+0000 7fa5751fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5700b5410 0x7fa570134a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:48.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.937+0000 7fa5751fb640 10 monclient(hunting): picked mon.noname-b con 0x7fa5700b5410 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:48.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.937+0000 7fa5751fb640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5701400a0 0x7fa570142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 10 monclient(hunting): picked mon.noname-c con 0x7fa5701400a0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa56ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5700b5410 0x7fa570134a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa570145c00 0x7fa570142d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 10 monclient(hunting): picked mon.noname-a con 0x7fa570145c00 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa567fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5701400a0 0x7fa570142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57014df40 con 0x7fa570145c00 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700a9ea0 con 0x7fa5700b5410 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:48.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57014e0c0 con 0x7fa5701400a0 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa5751fb640 10 monclient(hunting): authenticate will time out at 1467.101110s 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa56ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5700b5410 0x7fa570134a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:39102/0 (socket says 172.21.15.49:39102) 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa56ed9d640 1 -- 172.21.15.49:0/1674921047 learned_addr learned my addr 172.21.15.49:0/1674921047 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa56ed9d640 10 monclient(hunting): get_auth_request con 0x7fa5700b5410 auth_method 0 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa56ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa56ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.938+0000 7fa56ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.939+0000 7fa567fff640 10 monclient(hunting): get_auth_request con 0x7fa5701400a0 auth_method 0 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.939+0000 7fa567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.939+0000 7fa567fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.939+0000 7fa567fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.939+0000 7fa56ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.939+0000 7fa56ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:48.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.939+0000 7fa56ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:48.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.939+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:48.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.939+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:48.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.939+0000 7fa567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:48.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.942+0000 7fa56f59e640 1 --2- 172.21.15.49:0/1674921047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa570145c00 0x7fa570142d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:48.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.943+0000 7fa56ed9d640 10 monclient(hunting): handle_auth_done global_id 4165 payload 824 2024-05-15T12:31:48.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.943+0000 7fa56ed9d640 1 -- 172.21.15.49:0/1674921047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5701400a0 msgr2=0x7fa570142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:48.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.943+0000 7fa56ed9d640 1 --2- 172.21.15.49:0/1674921047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5701400a0 0x7fa570142800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:48.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.943+0000 7fa56ed9d640 1 -- 172.21.15.49:0/1674921047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa570145c00 msgr2=0x7fa570142d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:48.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.943+0000 7fa56ed9d640 1 --2- 172.21.15.49:0/1674921047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa570145c00 0x7fa570142d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:48.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.943+0000 7fa56ed9d640 10 monclient: _finish_hunting 0 2024-05-15T12:31:48.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.943+0000 7fa56ed9d640 1 monclient: found mon.noname-b 2024-05-15T12:31:48.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.943+0000 7fa56ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:48.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.943+0000 7fa56ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:31:48.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.943+0000 7fa56ed9d640 1 -- 172.21.15.49:0/1674921047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa570142100 con 0x7fa5700b5410 2024-05-15T12:31:48.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.947+0000 7fa56ed9d640 1 --2- 172.21.15.49:0/1674921047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5700b5410 0x7fa570134a80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa56000c4a0 tx=0x7fa56000c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:48.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.947+0000 7fa5677fe640 10 client.?.objecter ms_handle_connect 0x7fa5700b5410 2024-05-15T12:31:48.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.947+0000 7fa5677fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:48.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 1 -- 172.21.15.49:0/1674921047 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560017070 con 0x7fa5700b5410 2024-05-15T12:31:48.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:48.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:48.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 10 monclient: dump: 2024-05-15T12:31:48.946 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 10 monclient: _finish_auth 0 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 10 monclient: _check_auth_tickets 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 1 -- 172.21.15.49:0/1674921047 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa560005ce0 con 0x7fa5700b5410 2024-05-15T12:31:48.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5751fb640 5 monclient: authenticate success, global_id 4165 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 1 -- 172.21.15.49:0/1674921047 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560007270 con 0x7fa5700b5410 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa5677fe640 10 monclient: dump: 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:48.948 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:31:48.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.956+0000 7fa56dd9b640 4 set_mon_vals no callback set 2024-05-15T12:31:48.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.957+0000 7fa5751fb640 10 monclient: _renew_subs 2024-05-15T12:31:48.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.957+0000 7fa5751fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:48.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.957+0000 7fa5751fb640 1 -- 172.21.15.49:0/1674921047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5701423e0 con 0x7fa5700b5410 2024-05-15T12:31:48.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.957+0000 7fa5751fb640 10 client.4165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:48.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.957+0000 7fa5751fb640 10 monclient: _renew_subs 2024-05-15T12:31:48.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.957+0000 7fa5751fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:48.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.957+0000 7fa5751fb640 1 -- 172.21.15.49:0/1674921047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa570140500 con 0x7fa5700b5410 2024-05-15T12:31:48.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.957+0000 7fa5751fb640 1 librados: init done 2024-05-15T12:31:48.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.958+0000 7fa5751fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:48.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.958+0000 7fa5751fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:48.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.958+0000 7fa5751fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:48.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:48.958+0000 7fa5751fb640 1 -- 172.21.15.49:0/1674921047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57004f0e0 con 0x7fa5700b5410 2024-05-15T12:31:49.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.203+0000 7fa5677fe640 1 -- 172.21.15.49:0/1674921047 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 2) v1 ==== 44914+0+0 (secure 0 0 0) 0x7fa56001a070 con 0x7fa5700b5410 2024-05-15T12:31:49.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.214+0000 7fa5677fe640 1 -- 172.21.15.49:0/1674921047 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa560010e50 con 0x7fa5700b5410 2024-05-15T12:31:49.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.214+0000 7fa5677fe640 10 client.4165.objecter ms_dispatch 0x7fa570000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:31:49.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.214+0000 7fa5677fe640 3 client.4165.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:31:49.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.214+0000 7fa5677fe640 3 client.4165.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:31:49.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.214+0000 7fa5677fe640 20 client.4165.objecter dump_active .. 0 homeless 2024-05-15T12:31:49.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.214+0000 7fa5677fe640 1 -- 172.21.15.49:0/1674921047 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa560007410 con 0x7fa5700b5410 2024-05-15T12:31:49.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.214+0000 7fa5677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:49.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.214+0000 7fa5677fe640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:49.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.346+0000 7fa5751fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:31:49.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.346+0000 7fa5751fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:31:49.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.346+0000 7fa5751fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:49.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.346+0000 7fa5751fb640 1 -- 172.21.15.49:0/1674921047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa570146ad0 con 0x7fa5700b5410 2024-05-15T12:31:49.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.446+0000 7fa5677fe640 1 -- 172.21.15.49:0/1674921047 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fa56001a760 con 0x7fa5700b5410 2024-05-15T12:31:49.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.446+0000 7fa5677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:31:49.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.446+0000 7fa5677fe640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:49.437 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:31:49.437 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":10,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:31:48.140010+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 10 librados: watch_flush enter 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 10 librados: watch_flush exit 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 20 client.4165.objecter shutdown clearing up homeless session... 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 10 client.4165.objecter successfully canceled tick 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 10 monclient: shutdown 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 1 -- 172.21.15.49:0/1674921047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5700b5410 msgr2=0x7fa570134a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 1 --2- 172.21.15.49:0/1674921047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5700b5410 0x7fa570134a80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa56000c4a0 tx=0x7fa56000c960 comp rx=0 tx=0).stop 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 1 -- 172.21.15.49:0/1674921047 shutdown_connections 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 1 --2- 172.21.15.49:0/1674921047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa570145c00 0x7fa570142d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 1 --2- 172.21.15.49:0/1674921047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5701400a0 0x7fa570142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 1 --2- 172.21.15.49:0/1674921047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5700b5410 0x7fa570134a80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:49.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 1 -- 172.21.15.49:0/1674921047 >> 172.21.15.49:0/1674921047 conn(0x7fa570066f00 msgr2=0x7fa57013f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:49.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 1 -- 172.21.15.49:0/1674921047 shutdown_connections 2024-05-15T12:31:49.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 1 -- 172.21.15.49:0/1674921047 wait complete. 2024-05-15T12:31:49.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:31:49.448+0000 7fa5751fb640 1 librados: shutdown 2024-05-15T12:31:49.455 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-15T12:31:49.455 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-15T12:31:49.948 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:49.958+0000 7f838f5c1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-15T12:31:50.197 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T12:31:50.207+0000 7f838f5c1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-15T12:31:50.260 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:31:50.390 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.399+0000 7f217e8c1640 10 monclient: get_monmap_and_config 2024-05-15T12:31:50.390 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.399+0000 7f217e8c1640 10 monclient: build_initial_monmap 2024-05-15T12:31:50.390 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.399+0000 7f217e8c1640 10 monclient: monmap: 2024-05-15T12:31:50.390 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:31:50.390 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:50.390 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:31:50.401335+0000 2024-05-15T12:31:50.390 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:31:50.401335+0000 2024-05-15T12:31:50.390 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:50.390 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:50.390 INFO:teuthology.misc.health.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:50.391 INFO:teuthology.misc.health.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:50.391 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:50.391 INFO:teuthology.misc.health.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:50.391 INFO:teuthology.misc.health.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:50.391 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:50.391 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.401+0000 7f217e8c1640 1 Processor -- start 2024-05-15T12:31:50.391 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.401+0000 7f217e8c1640 1 -- start start 2024-05-15T12:31:50.392 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.401+0000 7f217e8c1640 10 monclient: init 2024-05-15T12:31:50.393 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:50.393 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f217813ffa0 0x7f21780728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f217813ffa0 addr v2:172.21.15.88:3300/0 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f21780b23b0 0x7f21781426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f21780b23b0 addr v2:172.21.15.49:3300/0 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2178142be0 addr v1:172.21.15.88:6789/0 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21780ab160 con 0x7f2178142be0 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2178143000 con 0x7f21780b23b0 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21780ab440 con 0x7f217813ffa0 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:50.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.402+0000 7f217e8c1640 10 monclient(hunting): authenticate will time out at 1468.564719s 2024-05-15T12:31:50.409 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.419+0000 7f217ce5e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2178142be0 0x7f21780b2970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:45442/0 (socket says 172.21.15.49:45442) 2024-05-15T12:31:50.409 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.419+0000 7f217ce5e640 1 -- 172.21.15.49:0/3886101364 learned_addr learned my addr 172.21.15.49:0/3886101364 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:31:50.410 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.420+0000 7f2177fff640 1 --2- 172.21.15.49:0/3886101364 >> v2:172.21.15.88:3300/0 conn(0x7f217813ffa0 0x7f21780728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:50.411 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.421+0000 7f2177fff640 10 monclient(hunting): get_auth_request con 0x7f217813ffa0 auth_method 0 2024-05-15T12:31:50.411 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.421+0000 7f2177fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:50.412 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.421+0000 7f2177fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:50.412 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.421+0000 7f2177fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:50.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.423+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:50.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.423+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:50.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.423+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:50.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.426+0000 7f2177fff640 10 monclient(hunting): handle_auth_done global_id 4168 payload 274 2024-05-15T12:31:50.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.426+0000 7f2177fff640 1 -- 172.21.15.49:0/3886101364 >> v2:172.21.15.49:3300/0 conn(0x7f21780b23b0 msgr2=0x7f21781426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:31:50.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.426+0000 7f2177fff640 1 --2- 172.21.15.49:0/3886101364 >> v2:172.21.15.49:3300/0 conn(0x7f21780b23b0 0x7f21781426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:50.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.426+0000 7f2177fff640 1 -- 172.21.15.49:0/3886101364 >> v1:172.21.15.88:6789/0 conn(0x7f2178142be0 legacy=0x7f21780b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:50.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.426+0000 7f2177fff640 10 monclient: _finish_hunting 0 2024-05-15T12:31:50.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.426+0000 7f2177fff640 1 monclient: found mon.noname-c 2024-05-15T12:31:50.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.426+0000 7f2177fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:50.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.426+0000 7f2177fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:31:50.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.426+0000 7f2177fff640 1 -- 172.21.15.49:0/3886101364 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21780a9ae0 con 0x7f217813ffa0 2024-05-15T12:31:50.421 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.431+0000 7f2177fff640 1 --2- 172.21.15.49:0/3886101364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f217813ffa0 0x7f21780728b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f21780b5b10 tx=0x7f216802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9aeb24c49bcbda66 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:50.425 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.435+0000 7f2176ffd640 1 -- 172.21.15.49:0/3886101364 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680315a0 con 0x7f217813ffa0 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.435+0000 7f2176ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.435+0000 7f2176ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.435+0000 7f2176ffd640 10 monclient: dump: 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:50.426 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:50.427 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.435+0000 7f2176ffd640 10 monclient: _finish_auth 0 2024-05-15T12:31:50.427 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.435+0000 7f2176ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:31:50.427 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.435+0000 7f2176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:50.427 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.435+0000 7f217e8c1640 5 monclient: authenticate success, global_id 4168 2024-05-15T12:31:50.427 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.435+0000 7f217e8c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:50.428 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.437+0000 7f2176ffd640 1 -- 172.21.15.49:0/3886101364 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2168037070 con 0x7f217813ffa0 2024-05-15T12:31:50.428 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.437+0000 7f2176ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:50.428 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.437+0000 7f2176ffd640 1 -- 172.21.15.49:0/3886101364 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168031960 con 0x7f217813ffa0 2024-05-15T12:31:50.428 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.437+0000 7f2176ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:50.428 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.437+0000 7f2176ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:50.428 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.437+0000 7f2176ffd640 10 monclient: dump: 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.438+0000 7f217e8c1640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:50.429 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.438+0000 7f217e8c1640 4 set_mon_vals no callback set 2024-05-15T12:31:50.431 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.438+0000 7f217e8c1640 10 monclient: shutdown 2024-05-15T12:31:50.431 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.438+0000 7f217e8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:50.431 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.438+0000 7f217e8c1640 1 -- 172.21.15.49:0/3886101364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f217813ffa0 msgr2=0x7f21780728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:50.431 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.438+0000 7f217e8c1640 1 --2- 172.21.15.49:0/3886101364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f217813ffa0 0x7f21780728b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f21780b5b10 tx=0x7f216802b840 comp rx=0 tx=0).stop 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.439+0000 7f217e8c1640 1 -- 172.21.15.49:0/3886101364 shutdown_connections 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.439+0000 7f217e8c1640 1 --2- 172.21.15.49:0/3886101364 >> v2:172.21.15.49:3300/0 conn(0x7f21780b23b0 0x7f21781426a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.439+0000 7f217e8c1640 1 --2- 172.21.15.49:0/3886101364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f217813ffa0 0x7f21780728b0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f21780b5b10 tx=0x7f216802b840 comp rx=0 tx=0).stop 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.439+0000 7f217e8c1640 1 -- 172.21.15.49:0/3886101364 >> 172.21.15.49:0/3886101364 conn(0x7f21780b8390 msgr2=0x7f21780b87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.439+0000 7f217e8c1640 1 -- 172.21.15.49:0/3886101364 shutdown_connections 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.439+0000 7f217e8c1640 1 -- 172.21.15.49:0/3886101364 wait complete. 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.440+0000 7f217e8c1640 10 monclient: build_initial_monmap 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.440+0000 7f217e8c1640 10 monclient: monmap: 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:last_changed 0.000000 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:created 0.000000 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:50.432 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:50.433 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:50.433 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:50.433 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:50.433 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.440+0000 7f217e8c1640 1 librados: starting msgr at 2024-05-15T12:31:50.433 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.440+0000 7f217e8c1640 1 librados: starting objecter 2024-05-15T12:31:50.433 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.441+0000 7f217e8c1640 1 Processor -- start 2024-05-15T12:31:50.433 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.441+0000 7f217e8c1640 1 -- start start 2024-05-15T12:31:50.433 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.441+0000 7f217e8c1640 1 librados: setting wanted keys 2024-05-15T12:31:50.433 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.441+0000 7f217e8c1640 1 librados: calling monclient init 2024-05-15T12:31:50.433 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.441+0000 7f217e8c1640 10 monclient: init 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21780b23b0 0x7f2178138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f21780b23b0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2178142be0 0x7f21781ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f2178142be0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2178138090 0x7f2178138520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f2178138090 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21780aa180 con 0x7f21780b23b0 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21780aa000 con 0x7f2178138090 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:50.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21780b3080 con 0x7f2178142be0 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.442+0000 7f217e8c1640 10 monclient(hunting): authenticate will time out at 1468.605163s 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.443+0000 7f217ce5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2178138090 0x7f2178138520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.443+0000 7f217ce5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2178138090 0x7f2178138520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:39132/0 (socket says 172.21.15.49:39132) 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.443+0000 7f217ce5e640 1 -- 172.21.15.49:0/2588484047 learned_addr learned my addr 172.21.15.49:0/2588484047 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.443+0000 7f217ce5e640 10 monclient(hunting): get_auth_request con 0x7f2178138090 auth_method 0 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.443+0000 7f217ce5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.443+0000 7f217ce5e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.443+0000 7f217ce5e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.444+0000 7f217ce5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.444+0000 7f217ce5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.444+0000 7f217ce5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:50.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.445+0000 7f21777fe640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2178142be0 0x7f21781ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:50.441 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f21777fe640 10 monclient(hunting): get_auth_request con 0x7f2178142be0 auth_method 0 2024-05-15T12:31:50.441 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f21777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:50.441 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f21777fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:50.441 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f21777fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:50.441 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f217ce5e640 10 monclient(hunting): handle_auth_done global_id 4171 payload 824 2024-05-15T12:31:50.441 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f217ce5e640 1 -- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2178142be0 msgr2=0x7f21781ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:50.441 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f217ce5e640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2178142be0 0x7f21781ceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:50.441 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f217ce5e640 1 -- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21780b23b0 msgr2=0x7f2178138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:50.441 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f217ce5e640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21780b23b0 0x7f2178138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:50.441 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f217ce5e640 10 monclient: _finish_hunting 0 2024-05-15T12:31:50.442 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f217ce5e640 1 monclient: found mon.noname-b 2024-05-15T12:31:50.442 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f217ce5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:50.442 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f217ce5e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:31:50.442 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.450+0000 7f217ce5e640 1 -- 172.21.15.49:0/2588484047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21781ce630 con 0x7f2178138090 2024-05-15T12:31:50.450 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.460+0000 7f217ce5e640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2178138090 0x7f2178138520 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f217000c9f0 tx=0x7f217000ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:50.451 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.460+0000 7f2174ff9640 10 client.?.objecter ms_handle_connect 0x7f2178138090 2024-05-15T12:31:50.451 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.460+0000 7f2174ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:50.461 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 1 -- 172.21.15.49:0/2588484047 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2170017070 con 0x7f2178138090 2024-05-15T12:31:50.461 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 10 monclient: dump: 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:50.462 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 10 monclient: _finish_auth 0 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 10 monclient: _check_auth_tickets 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 1 -- 172.21.15.49:0/2588484047 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217000ab70 con 0x7f2178138090 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 1 -- 172.21.15.49:0/2588484047 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2170012400 con 0x7f2178138090 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f2174ff9640 10 monclient: dump: 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:50.463 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f21767fc640 4 set_mon_vals no callback set 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f217e8c1640 5 monclient: authenticate success, global_id 4171 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f217e8c1640 10 monclient: _renew_subs 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f217e8c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.469+0000 7f217e8c1640 1 -- 172.21.15.49:0/2588484047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21781cf6e0 con 0x7f2178138090 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.470+0000 7f217e8c1640 10 client.4171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.470+0000 7f217e8c1640 10 monclient: _renew_subs 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.470+0000 7f217e8c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.470+0000 7f217e8c1640 1 -- 172.21.15.49:0/2588484047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2178138a60 con 0x7f2178138090 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.470+0000 7f217e8c1640 1 librados: init done 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.471+0000 7f217e8c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.471+0000 7f217e8c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:50.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.471+0000 7f217e8c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:50.465 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.471+0000 7f217e8c1640 1 -- 172.21.15.49:0/2588484047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2178068270 con 0x7f2178138090 2024-05-15T12:31:50.924 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.933+0000 7f2174ff9640 1 -- 172.21.15.49:0/2588484047 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 3) v1 ==== 89616+0+0 (secure 0 0 0) 0x7f217001a070 con 0x7f2178138090 2024-05-15T12:31:50.924 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.934+0000 7f2174ff9640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f215806cd00 0x7f215806f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:50.924 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.934+0000 7f2177fff640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f215806cd00 0x7f215806f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:50.924 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.934+0000 7f2177fff640 10 monclient: get_auth_request con 0x7f215806cd00 auth_method 0 2024-05-15T12:31:50.925 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.935+0000 7f2174ff9640 10 client.4171.objecter ms_handle_connect 0x7f215806cd00 2024-05-15T12:31:50.925 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.935+0000 7f2177fff640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f215806cd00 0x7f215806f1c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2168036000 tx=0x7f2168030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:50.938 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.947+0000 7f2174ff9640 1 -- 172.21.15.49:0/2588484047 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f21700576f0 con 0x7f2178138090 2024-05-15T12:31:50.938 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.948+0000 7f2174ff9640 10 client.4171.objecter ms_dispatch 0x7f2178000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:31:50.938 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.948+0000 7f2174ff9640 3 client.4171.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:31:50.938 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.948+0000 7f2174ff9640 3 client.4171.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:31:50.938 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.948+0000 7f2174ff9640 20 client.4171.objecter dump_active .. 0 homeless 2024-05-15T12:31:50.965 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.974+0000 7f2174ff9640 1 -- 172.21.15.49:0/2588484047 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2170058a60 con 0x7f2178138090 2024-05-15T12:31:50.965 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.974+0000 7f2174ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:50.965 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:50.975+0000 7f2174ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:51.207 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.216+0000 7f2174ff9640 1 -- 172.21.15.49:0/2588484047 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f21700550d0 con 0x7f2178138090 2024-05-15T12:31:51.343 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.353+0000 7f217e8c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:31:51.344 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.353+0000 7f217e8c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:31:51.344 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.353+0000 7f217e8c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:51.344 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.353+0000 7f217e8c1640 1 -- 172.21.15.49:0/2588484047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21780aa3f0 con 0x7f2178138090 2024-05-15T12:31:51.476 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.485+0000 7f2174ff9640 1 -- 172.21.15.49:0/2588484047 <== mon.1 v2:172.21.15.88:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f217003e770 con 0x7f2178138090 2024-05-15T12:31:51.476 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.486+0000 7f2174ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:31:51.476 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.486+0000 7f2174ff9640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:51.476 INFO:teuthology.misc.health.smithi049.stdout: 2024-05-15T12:31:51.476 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":10,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:31:48.140010+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:31:51.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.488+0000 7f217e8c1640 10 librados: watch_flush enter 2024-05-15T12:31:51.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 10 librados: watch_flush exit 2024-05-15T12:31:51.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 20 client.4171.objecter shutdown clearing up homeless session... 2024-05-15T12:31:51.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 10 client.4171.objecter successfully canceled tick 2024-05-15T12:31:51.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 -- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f215806cd00 msgr2=0x7f215806f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:51.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f215806cd00 0x7f215806f1c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2168036000 tx=0x7f2168030000 comp rx=0 tx=0).stop 2024-05-15T12:31:51.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 10 monclient: shutdown 2024-05-15T12:31:51.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:51.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 -- 172.21.15.49:0/2588484047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2178138090 msgr2=0x7f2178138520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:51.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2178138090 0x7f2178138520 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f217000c9f0 tx=0x7f217000ceb0 comp rx=0 tx=0).stop 2024-05-15T12:31:51.480 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 -- 172.21.15.49:0/2588484047 shutdown_connections 2024-05-15T12:31:51.480 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f215806cd00 0x7f215806f1c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:51.480 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2178138090 0x7f2178138520 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:51.480 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2178142be0 0x7f21781ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:51.480 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 --2- 172.21.15.49:0/2588484047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21780b23b0 0x7f2178138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:51.481 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 -- 172.21.15.49:0/2588484047 >> 172.21.15.49:0/2588484047 conn(0x7f21780b8390 msgr2=0x7f21780b8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:51.481 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 -- 172.21.15.49:0/2588484047 shutdown_connections 2024-05-15T12:31:51.481 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 -- 172.21.15.49:0/2588484047 wait complete. 2024-05-15T12:31:51.481 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:51.489+0000 7f217e8c1640 1 librados: shutdown 2024-05-15T12:31:51.494 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-15T12:31:55.876 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:07.238 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:55.939 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:07.300 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:55.939 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:07.303 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:55.982 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:07.347 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.001 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:07.366 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.058 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:07.423 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.376 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:07.718 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.437 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:07.779 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.489 INFO:tasks.ceph.osd.6.smithi088.stderr:2024-05-15T12:31:56.494+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:31:56.509 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:07.847 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.553 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:31:56.559+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:31:56.579 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:07.918 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.604 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-15T12:31:56.609+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:31:56.774 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:08.123 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.821 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:08.182 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.837 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:08.186 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.884 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:08.245 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.952 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:08.305 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:56.964 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:08.326 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.000 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:31:57.006+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:31:57.013 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:08.367 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.039 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:08.403 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.099 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:08.463 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.122 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:31:57.127+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:31:57.335 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:08.697 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.409 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:08.774 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.411 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:31:57.414+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:31:57.435 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:31:57.440+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:31:57.474 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:08.839 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.480 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:08.822 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.587 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:31:57.591+0000 5dce500 -1 Falling back to public interface 2024-05-15T12:31:57.630 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:08.969 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.648 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:09.010 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.763 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:09.127 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.797 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:09.162 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.854 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:09.196 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.916 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:09.278 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.926 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:09.275 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:57.950 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:09.310 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.009 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:09.348 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.051 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:09.415 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.064 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:09.429 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.130 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:09.484 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.184 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:09.546 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.205 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:09.548 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.297 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:09.647 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.300 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:31:58.325 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:09.686 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.328 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:09.693 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.333 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:09.672 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.344 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:09.709 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.405+0000 7f6578ec8640 10 monclient: get_monmap_and_config 2024-05-15T12:31:58.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.405+0000 7f6578ec8640 10 monclient: build_initial_monmap 2024-05-15T12:31:58.394 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.405+0000 7f6578ec8640 10 monclient: monmap: 2024-05-15T12:31:58.394 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:31:58.394 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:58.394 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:31:58.405902+0000 2024-05-15T12:31:58.394 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:31:58.405902+0000 2024-05-15T12:31:58.395 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:58.395 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:58.395 INFO:teuthology.misc.health.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:31:58.395 INFO:teuthology.misc.health.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:31:58.395 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:31:58.395 INFO:teuthology.misc.health.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:31:58.395 INFO:teuthology.misc.health.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:31:58.395 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:58.396 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 1 Processor -- start 2024-05-15T12:31:58.396 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 1 -- start start 2024-05-15T12:31:58.396 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 10 monclient: init 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65740b51e0 addr v1:172.21.15.88:6789/0 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b23d0 0x7f657413ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 10 monclient(hunting): picked mon.noname-b con 0x7f65740b23d0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6574142770 addr v1:172.21.15.49:6789/0 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65740aa230 con 0x7f6574142770 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6574143310 con 0x7f65740b51e0 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65740b55c0 con 0x7f65740b23d0 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.406+0000 7f6578ec8640 10 monclient(hunting): authenticate will time out at 1476.568383s 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.407+0000 7f6571d9b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b23d0 0x7f657413ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:58.397 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.407+0000 7f657259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f65740b51e0 0x7f657413f8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:46182/0 (socket says 172.21.15.49:46182) 2024-05-15T12:31:58.398 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.407+0000 7f657259c640 1 -- 172.21.15.49:0/638357518 learned_addr learned my addr 172.21.15.49:0/638357518 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:31:58.398 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.407+0000 7f6571d9b640 10 monclient(hunting): get_auth_request con 0x7f65740b23d0 auth_method 0 2024-05-15T12:31:58.398 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.407+0000 7f6571d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:58.398 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.407+0000 7f6571d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:58.398 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.407+0000 7f6571d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:58.399 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.408+0000 7f6571d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:58.399 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.408+0000 7f6571d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:58.399 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.408+0000 7f6571d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:58.399 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.409+0000 7f657159a640 1 -- 172.21.15.49:0/638357518 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800404071 0 0) 0x7f6574143310 con 0x7f65740b51e0 2024-05-15T12:31:58.399 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.409+0000 7f657159a640 10 monclient(hunting): _negotiate 2024-05-15T12:31:58.399 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.409+0000 7f657159a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.409+0000 7f657159a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.409+0000 7f657159a640 10 monclient(hunting): my global_id is 4198 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.409+0000 7f657159a640 1 -- 172.21.15.49:0/638357518 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f655c003520 con 0x7f65740b51e0 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.410+0000 7f6571d9b640 10 monclient(hunting): handle_auth_done global_id 4163 payload 274 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.410+0000 7f6571d9b640 1 -- 172.21.15.49:0/638357518 >> v1:172.21.15.88:6789/0 conn(0x7f65740b51e0 legacy=0x7f657413f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.410+0000 7f657159a640 1 -- 172.21.15.49:0/638357518 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318429498 0 0) 0x7f65740aa230 con 0x7f6574142770 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.410+0000 7f6571d9b640 1 -- 172.21.15.49:0/638357518 >> v1:172.21.15.49:6789/0 conn(0x7f6574142770 legacy=0x7f6574142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.410+0000 7f6571d9b640 10 monclient: _finish_hunting 0 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.410+0000 7f6571d9b640 1 monclient: found mon.noname-b 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.410+0000 7f6571d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.410+0000 7f6571d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.410+0000 7f6571d9b640 1 -- 172.21.15.49:0/638357518 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65740b3a80 con 0x7f65740b23d0 2024-05-15T12:31:58.400 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.410+0000 7f657159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-05-15T12:31:58.401 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.412+0000 7f6571d9b640 1 --2- 172.21.15.49:0/638357518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b23d0 0x7f657413ffc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6568009800 tx=0x7f656802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46c7388967860372 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 1 -- 172.21.15.49:0/638357518 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568034070 con 0x7f65740b23d0 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 10 monclient: dump: 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:58.405 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 10 monclient: _finish_auth 0 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 10 monclient: _check_auth_tickets 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 1 -- 172.21.15.49:0/638357518 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6568030880 con 0x7f65740b23d0 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f6578ec8640 5 monclient: authenticate success, global_id 4163 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f6578ec8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f6578ec8640 10 monclient: get_monmap_and_config success 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f6578ec8640 4 set_mon_vals no callback set 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 1 -- 172.21.15.49:0/638357518 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568030c50 con 0x7f65740b23d0 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.415+0000 7f657159a640 10 monclient: dump: 2024-05-15T12:31:58.406 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.416+0000 7f6578ec8640 10 monclient: shutdown 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.416+0000 7f6578ec8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.416+0000 7f6578ec8640 1 -- 172.21.15.49:0/638357518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b23d0 msgr2=0x7f657413ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:58.407 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.416+0000 7f6578ec8640 1 --2- 172.21.15.49:0/638357518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b23d0 0x7f657413ffc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6568009800 tx=0x7f656802b9c0 comp rx=0 tx=0).stop 2024-05-15T12:31:58.411 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.416+0000 7f6578ec8640 1 -- 172.21.15.49:0/638357518 shutdown_connections 2024-05-15T12:31:58.411 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.416+0000 7f6578ec8640 1 --2- 172.21.15.49:0/638357518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b23d0 0x7f657413ffc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:58.411 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.416+0000 7f6578ec8640 1 -- 172.21.15.49:0/638357518 >> 172.21.15.49:0/638357518 conn(0x7f6574067020 msgr2=0x7f6574067430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:58.411 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.416+0000 7f6578ec8640 1 -- 172.21.15.49:0/638357518 shutdown_connections 2024-05-15T12:31:58.411 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.416+0000 7f6578ec8640 1 -- 172.21.15.49:0/638357518 wait complete. 2024-05-15T12:31:58.411 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.417+0000 7f6578ec8640 10 monclient: build_initial_monmap 2024-05-15T12:31:58.411 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.417+0000 7f6578ec8640 10 monclient: monmap: 2024-05-15T12:31:58.411 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:last_changed 0.000000 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:created 0.000000 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.417+0000 7f6578ec8640 1 librados: starting msgr at 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.417+0000 7f6578ec8640 1 librados: starting objecter 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.417+0000 7f6578ec8640 1 Processor -- start 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.417+0000 7f6578ec8640 1 -- start start 2024-05-15T12:31:58.412 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.417+0000 7f6578ec8640 1 librados: setting wanted keys 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.417+0000 7f6578ec8640 1 librados: calling monclient init 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.417+0000 7f6578ec8640 10 monclient: init 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 10 monclient: _reopen_session rank -1 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740b23d0 0x7f6574135510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 10 monclient(hunting): picked mon.noname-b con 0x7f65740b23d0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b51e0 0x7f6574136f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 10 monclient(hunting): picked mon.noname-c con 0x7f65740b51e0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6574142770 0x7f65740b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 10 monclient(hunting): picked mon.noname-a con 0x7f6574142770 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657414e190 con 0x7f6574142770 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657414e010 con 0x7f65740b23d0 2024-05-15T12:31:58.413 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657414e310 con 0x7f65740b51e0 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 10 monclient(hunting): _renew_subs 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6578ec8640 10 monclient(hunting): authenticate will time out at 1476.579824s 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f6572d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6574142770 0x7f65740b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.418+0000 7f657259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740b23d0 0x7f6574135510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6571d9b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b51e0 0x7f6574136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f657259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740b23d0 0x7f6574135510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:53418/0 (socket says 172.21.15.49:53418) 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f657259c640 1 -- 172.21.15.49:0/1138981926 learned_addr learned my addr 172.21.15.49:0/1138981926 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f657259c640 10 monclient(hunting): get_auth_request con 0x7f65740b23d0 auth_method 0 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f657259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f657259c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f657259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6572d9d640 10 monclient(hunting): get_auth_request con 0x7f6574142770 auth_method 0 2024-05-15T12:31:58.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6572d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6572d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6572d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6571d9b640 10 monclient(hunting): get_auth_request con 0x7f65740b51e0 auth_method 0 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6571d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6571d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6571d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f657259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f657259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f657259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6572d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6572d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6572d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6571d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:31:58.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6571d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.419+0000 7f6571d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.422+0000 7f657259c640 10 monclient(hunting): handle_auth_done global_id 4201 payload 824 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.422+0000 7f657259c640 1 -- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b51e0 msgr2=0x7f6574136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.422+0000 7f657259c640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b51e0 0x7f6574136f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.422+0000 7f657259c640 1 -- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6574142770 msgr2=0x7f65740b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.422+0000 7f657259c640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6574142770 0x7f65740b5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.422+0000 7f657259c640 10 monclient: _finish_hunting 0 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.422+0000 7f657259c640 1 monclient: found mon.noname-b 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.422+0000 7f657259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.422+0000 7f657259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:31:58.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.422+0000 7f657259c640 1 -- 172.21.15.49:0/1138981926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6574136260 con 0x7f65740b23d0 2024-05-15T12:31:58.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.424+0000 7f657259c640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740b23d0 0x7f6574135510 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f656400c500 tx=0x7f656400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:58.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.424+0000 7f6562ffd640 10 client.?.objecter ms_handle_connect 0x7f65740b23d0 2024-05-15T12:31:58.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.424+0000 7f6562ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:31:58.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 1 -- 172.21.15.49:0/1138981926 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564017070 con 0x7f65740b23d0 2024-05-15T12:31:58.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:58.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:31:58.418 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 10 monclient: dump: 2024-05-15T12:31:58.418 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:31:58.418 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:58.418 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 10 monclient: _finish_auth 0 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 1 -- 172.21.15.49:0/1138981926 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6564005ce0 con 0x7f65740b23d0 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6578ec8640 5 monclient: authenticate success, global_id 4201 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6578ec8640 10 monclient: _renew_subs 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6578ec8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6578ec8640 1 -- 172.21.15.49:0/1138981926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6574136540 con 0x7f65740b23d0 2024-05-15T12:31:58.419 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 1 -- 172.21.15.49:0/1138981926 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564007330 con 0x7f65740b23d0 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6562ffd640 10 monclient: dump: 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6570d99640 4 set_mon_vals no callback set 2024-05-15T12:31:58.420 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6578ec8640 10 client.4201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:31:58.421 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6578ec8640 10 monclient: _renew_subs 2024-05-15T12:31:58.421 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6578ec8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:58.421 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6578ec8640 1 -- 172.21.15.49:0/1138981926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6574136b00 con 0x7f65740b23d0 2024-05-15T12:31:58.421 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.428+0000 7f6578ec8640 1 librados: init done 2024-05-15T12:31:58.421 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.429+0000 7f6578ec8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:58.421 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.429+0000 7f6578ec8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:58.421 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.429+0000 7f6578ec8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:58.421 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.429+0000 7f6578ec8640 1 -- 172.21.15.49:0/1138981926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657404f0e0 con 0x7f65740b23d0 2024-05-15T12:31:58.435 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.446+0000 7f6562ffd640 1 -- 172.21.15.49:0/1138981926 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f656401a070 con 0x7f65740b23d0 2024-05-15T12:31:58.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.447+0000 7f6562ffd640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f655406ce20 0x7f655406f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:31:58.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.447+0000 7f6571d9b640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f655406ce20 0x7f655406f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:31:58.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.447+0000 7f6571d9b640 10 monclient: get_auth_request con 0x7f655406ce20 auth_method 0 2024-05-15T12:31:58.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.447+0000 7f6571d9b640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f655406ce20 0x7f655406f2e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6568036000 tx=0x7f656803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:31:58.436 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.447+0000 7f6562ffd640 10 client.4201.objecter ms_handle_connect 0x7f655406ce20 2024-05-15T12:31:58.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.448+0000 7f6562ffd640 1 -- 172.21.15.49:0/1138981926 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6564056ff0 con 0x7f65740b23d0 2024-05-15T12:31:58.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.448+0000 7f6562ffd640 10 client.4201.objecter ms_dispatch 0x7f6574000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:31:58.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.448+0000 7f6562ffd640 3 client.4201.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:31:58.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.448+0000 7f6562ffd640 3 client.4201.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:31:58.437 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.448+0000 7f6562ffd640 20 client.4201.objecter dump_active .. 0 homeless 2024-05-15T12:31:58.447 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:09.809 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.494 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:09.836 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.522 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:09.876 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.594 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:09.959 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.598 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:09.937 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.635 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:09.999 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.645 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:10.006 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.664 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:10.014 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.677 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.688+0000 7f6562ffd640 1 -- 172.21.15.49:0/1138981926 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6564021020 con 0x7f65740b23d0 2024-05-15T12:31:58.677 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.688+0000 7f6562ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:31:58.678 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.688+0000 7f6562ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:31:58.714 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:10.076 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.785 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:10.128 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.840 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:10.194 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.866 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:10.205 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.869 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:10.234 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.870 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.881+0000 7f6578ec8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:31:58.870 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.881+0000 7f6578ec8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:31:58.870 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.881+0000 7f6578ec8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:31:58.870 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.881+0000 7f6578ec8640 1 -- 172.21.15.49:0/1138981926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f657406ae10 con 0x7f65740b23d0 2024-05-15T12:31:58.888 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.899+0000 7f6562ffd640 1 -- 172.21.15.49:0/1138981926 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f6564058360 con 0x7f65740b23d0 2024-05-15T12:31:58.888 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.899+0000 7f6562ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:31:58.888 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.899+0000 7f6562ffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:31:58.888 INFO:teuthology.misc.health.smithi049.stdout: 2024-05-15T12:31:58.888 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":10,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:31:48.140010+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:31:58.891 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 10 librados: watch_flush enter 2024-05-15T12:31:58.891 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 10 librados: watch_flush exit 2024-05-15T12:31:58.891 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 20 client.4201.objecter shutdown clearing up homeless session... 2024-05-15T12:31:58.891 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 10 client.4201.objecter successfully canceled tick 2024-05-15T12:31:58.891 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 -- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f655406ce20 msgr2=0x7f655406f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:58.891 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f655406ce20 0x7f655406f2e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6568036000 tx=0x7f656803b040 comp rx=0 tx=0).stop 2024-05-15T12:31:58.891 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 10 monclient: shutdown 2024-05-15T12:31:58.891 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:31:58.891 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 -- 172.21.15.49:0/1138981926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740b23d0 msgr2=0x7f6574135510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:31:58.892 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740b23d0 0x7f6574135510 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f656400c500 tx=0x7f656400c9c0 comp rx=0 tx=0).stop 2024-05-15T12:31:58.892 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 -- 172.21.15.49:0/1138981926 shutdown_connections 2024-05-15T12:31:58.892 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6574142770 0x7f65740b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:58.892 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65740b51e0 0x7f6574136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:58.892 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f655406ce20 0x7f655406f2e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:58.892 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 --2- 172.21.15.49:0/1138981926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65740b23d0 0x7f6574135510 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:31:58.892 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 -- 172.21.15.49:0/1138981926 >> 172.21.15.49:0/1138981926 conn(0x7f6574067020 msgr2=0x7f6574142c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:31:58.892 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 -- 172.21.15.49:0/1138981926 shutdown_connections 2024-05-15T12:31:58.892 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 -- 172.21.15.49:0/1138981926 wait complete. 2024-05-15T12:31:58.892 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:31:58.901+0000 7f6578ec8640 1 librados: shutdown 2024-05-15T12:31:58.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-15T12:31:58.917 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:10.278 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.926 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:10.291 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.967 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:10.316 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:58.978 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:10.340 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.077 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:10.419 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.108 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:10.462 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.132 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:10.471 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.133 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:10.498 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.186 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:10.547 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.214 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:10.579 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.250 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:10.612 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.259 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:10.608 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.368 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:10.710 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.380 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:10.734 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.397 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:10.762 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.411 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:10.750 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.450 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:10.811 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.506 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:10.871 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.548 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:10.897 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.649 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:11.003 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.662 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:11.004 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.680 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:11.019 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.715 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:11.076 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.724 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:11.086 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.779 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:11.140 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.839 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:11.188 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.885 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:11.250 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.914 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:11.268 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.939 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:11.304 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.948 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:11.287 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.971 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:11.313 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:31:59.989 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:11.350 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.030 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:11.395 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.085 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:11.450 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.129 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:11.478 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.186 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:11.540 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.266 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:11.627 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.421 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:11.771 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.444 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:11.782 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.456 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:11.810 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.503 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:11.842 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.536 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:11.879 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.591 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:11.933 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.813 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:12.174 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.870 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:12.231 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.953 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:12.307 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:00.975 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:12.325 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:01.010 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:12.364 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:01.032 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:12.382 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:05.530 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:16.892 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:05.562 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:16.927 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:05.706 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:32:05.800 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.811+0000 7fedb6f0b640 10 monclient: get_monmap_and_config 2024-05-15T12:32:05.800 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.811+0000 7fedb6f0b640 10 monclient: build_initial_monmap 2024-05-15T12:32:05.800 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.811+0000 7fedb6f0b640 10 monclient: monmap: 2024-05-15T12:32:05.800 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:05.800 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:05.800 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:32:05.811594+0000 2024-05-15T12:32:05.800 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:32:05.811594+0000 2024-05-15T12:32:05.801 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:05.801 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:05.801 INFO:teuthology.misc.health.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:05.801 INFO:teuthology.misc.health.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:05.801 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:05.801 INFO:teuthology.misc.health.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:05.801 INFO:teuthology.misc.health.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:05.801 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.811+0000 7fedb6f0b640 1 Processor -- start 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.811+0000 7fedb6f0b640 1 -- start start 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 10 monclient: init 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fedb013ffa0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fedb00b23b0 0x7fedb01426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 10 monclient(hunting): picked mon.noname-a con 0x7fedb00b23b0 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fedb0142be0 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb00ab160 con 0x7fedb013ffa0 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb00ab440 con 0x7fedb0142be0 2024-05-15T12:32:05.803 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:05.804 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb0143000 con 0x7fedb00b23b0 2024-05-15T12:32:05.804 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:05.804 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.812+0000 7fedb6f0b640 10 monclient(hunting): authenticate will time out at 1483.974287s 2024-05-15T12:32:05.804 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.813+0000 7fedb54a8640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fedb0142be0 0x7fedb014cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:38652/0 (socket says 172.21.15.49:38652) 2024-05-15T12:32:05.804 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.813+0000 7fedb54a8640 1 -- 172.21.15.49:0/3595817018 learned_addr learned my addr 172.21.15.49:0/3595817018 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:05.804 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.813+0000 7fedaffff640 1 --2- 172.21.15.49:0/3595817018 >> v2:172.21.15.49:3300/0 conn(0x7fedb00b23b0 0x7fedb01426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:05.804 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.814+0000 7fedaffff640 10 monclient(hunting): get_auth_request con 0x7fedb00b23b0 auth_method 0 2024-05-15T12:32:05.804 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.814+0000 7fedaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:05.804 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.814+0000 7fedaffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:05.804 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.814+0000 7fedaffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:05.805 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.815+0000 7fedaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:05.805 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.815+0000 7fedaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:05.805 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.815+0000 7fedaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:05.806 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.816+0000 7fedaf7fe640 1 -- 172.21.15.49:0/3595817018 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030466983 0 0) 0x7fedb00ab440 con 0x7fedb0142be0 2024-05-15T12:32:05.806 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.816+0000 7fedaf7fe640 10 monclient(hunting): _negotiate 2024-05-15T12:32:05.806 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.816+0000 7fedaf7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:05.806 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.816+0000 7fedaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:05.806 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.816+0000 7fedaf7fe640 10 monclient(hunting): my global_id is 4204 2024-05-15T12:32:05.806 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.816+0000 7fedaf7fe640 1 -- 172.21.15.49:0/3595817018 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0003520 con 0x7fedb0142be0 2024-05-15T12:32:05.807 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.817+0000 7fedaffff640 10 monclient(hunting): handle_auth_done global_id 4200 payload 274 2024-05-15T12:32:05.807 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.817+0000 7fedaffff640 1 -- 172.21.15.49:0/3595817018 >> v1:172.21.15.88:6789/0 conn(0x7fedb0142be0 legacy=0x7fedb014cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:05.807 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.817+0000 7fedaffff640 1 -- 172.21.15.49:0/3595817018 >> v1:172.21.15.49:6789/0 conn(0x7fedb013ffa0 legacy=0x7fedb00728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:05.807 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.817+0000 7fedaffff640 10 monclient: _finish_hunting 0 2024-05-15T12:32:05.807 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.817+0000 7fedaffff640 1 monclient: found mon.noname-a 2024-05-15T12:32:05.807 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.817+0000 7fedaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:05.807 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.817+0000 7fedaffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:05.807 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.817+0000 7fedaffff640 1 -- 172.21.15.49:0/3595817018 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb00a9ae0 con 0x7fedb00b23b0 2024-05-15T12:32:05.811 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.822+0000 7fedaffff640 1 --2- 172.21.15.49:0/3595817018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedb00b23b0 0x7fedb01426a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0097a0 tx=0x7fed9c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b71b2c63d59af18 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:05.816 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 1 -- 172.21.15.49:0/3595817018 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c035020 con 0x7fedb00b23b0 2024-05-15T12:32:05.816 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:05.816 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 10 monclient: dump: 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:05.817 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 10 monclient: _finish_auth 0 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 1 -- 172.21.15.49:0/3595817018 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c031830 con 0x7fedb00b23b0 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 1 -- 172.21.15.49:0/3595817018 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c031c00 con 0x7fedb00b23b0 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedaf7fe640 10 monclient: dump: 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:05.818 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:05.819 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:05.819 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:05.819 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:05.819 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:05.819 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedb6f0b640 5 monclient: authenticate success, global_id 4200 2024-05-15T12:32:05.819 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedb6f0b640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:05.819 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.827+0000 7fedb6f0b640 4 set_mon_vals no callback set 2024-05-15T12:32:05.822 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.828+0000 7fedb6f0b640 10 monclient: shutdown 2024-05-15T12:32:05.822 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.828+0000 7fedb6f0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:05.822 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.828+0000 7fedb6f0b640 1 -- 172.21.15.49:0/3595817018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedb00b23b0 msgr2=0x7fedb01426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:05.822 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.828+0000 7fedb6f0b640 1 --2- 172.21.15.49:0/3595817018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedb00b23b0 0x7fedb01426a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0097a0 tx=0x7fed9c02b8d0 comp rx=0 tx=0).stop 2024-05-15T12:32:05.822 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.828+0000 7fedb6f0b640 1 -- 172.21.15.49:0/3595817018 shutdown_connections 2024-05-15T12:32:05.822 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.828+0000 7fedb6f0b640 1 --2- 172.21.15.49:0/3595817018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedb00b23b0 0x7fedb01426a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:05.822 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.828+0000 7fedb6f0b640 1 -- 172.21.15.49:0/3595817018 >> 172.21.15.49:0/3595817018 conn(0x7fedb00b8390 msgr2=0x7fedb00b87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.828+0000 7fedb6f0b640 1 -- 172.21.15.49:0/3595817018 shutdown_connections 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.828+0000 7fedb6f0b640 1 -- 172.21.15.49:0/3595817018 wait complete. 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.829+0000 7fedb6f0b640 10 monclient: build_initial_monmap 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.829+0000 7fedb6f0b640 10 monclient: monmap: 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:created 0.000000 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:05.823 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.829+0000 7fedb6f0b640 1 librados: starting msgr at 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.829+0000 7fedb6f0b640 1 librados: starting objecter 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.830+0000 7fedb6f0b640 1 Processor -- start 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.830+0000 7fedb6f0b640 1 -- start start 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.830+0000 7fedb6f0b640 1 librados: setting wanted keys 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.830+0000 7fedb6f0b640 1 librados: calling monclient init 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.830+0000 7fedb6f0b640 10 monclient: init 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.830+0000 7fedb6f0b640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.830+0000 7fedb6f0b640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.830+0000 7fedb6f0b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedb00b23b0 0x7fedb013cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.830+0000 7fedb6f0b640 10 monclient(hunting): picked mon.noname-a con 0x7fedb00b23b0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.830+0000 7fedb6f0b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedb013ffa0 0x7fedb0135320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 10 monclient(hunting): picked mon.noname-b con 0x7fedb013ffa0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedb0142be0 0x7fedb0135860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 10 monclient(hunting): picked mon.noname-c con 0x7fedb0142be0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:05.824 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb014e7d0 con 0x7fedb00b23b0 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb014e650 con 0x7fedb013ffa0 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb014e950 con 0x7fedb0142be0 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb6f0b640 10 monclient(hunting): authenticate will time out at 1483.992732s 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedaffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedb013ffa0 0x7fedb0135320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedb4ca7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedb00b23b0 0x7fedb013cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedaffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedb013ffa0 0x7fedb0135320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:55546/0 (socket says 172.21.15.49:55546) 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedaffff640 1 -- 172.21.15.49:0/2948515942 learned_addr learned my addr 172.21.15.49:0/2948515942 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedaffff640 10 monclient(hunting): get_auth_request con 0x7fedb013ffa0 auth_method 0 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedaffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:05.825 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.831+0000 7fedaffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb4ca7640 10 monclient(hunting): get_auth_request con 0x7fedb00b23b0 auth_method 0 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb4ca7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb4ca7640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb4ca7640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb54a8640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedb0142be0 0x7fedb0135860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb54a8640 10 monclient(hunting): get_auth_request con 0x7fedb0142be0 auth_method 0 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb54a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb54a8640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb54a8640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb4ca7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:05.826 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb4ca7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:05.827 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.832+0000 7fedb4ca7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:05.827 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.833+0000 7fedb54a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:05.827 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.833+0000 7fedb54a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:05.827 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.833+0000 7fedb54a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:05.828 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.835+0000 7fedaffff640 10 monclient(hunting): handle_auth_done global_id 4207 payload 824 2024-05-15T12:32:05.828 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.835+0000 7fedaffff640 1 -- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedb0142be0 msgr2=0x7fedb0135860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.835+0000 7fedaffff640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedb0142be0 0x7fedb0135860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.835+0000 7fedaffff640 1 -- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedb00b23b0 msgr2=0x7fedb013cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.835+0000 7fedaffff640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedb00b23b0 0x7fedb013cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.835+0000 7fedaffff640 10 monclient: _finish_hunting 0 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.835+0000 7fedaffff640 1 monclient: found mon.noname-b 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.835+0000 7fedaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.835+0000 7fedaffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.835+0000 7fedaffff640 1 -- 172.21.15.49:0/2948515942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb00ace70 con 0x7fedb013ffa0 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.837+0000 7fedaffff640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedb013ffa0 0x7fedb0135320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fed9c034040 tx=0x7fed9c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.837+0000 7fedad7fa640 10 client.?.objecter ms_handle_connect 0x7fedb013ffa0 2024-05-15T12:32:05.829 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.837+0000 7fedad7fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:05.833 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 1 -- 172.21.15.49:0/2948515942 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c040070 con 0x7fedb013ffa0 2024-05-15T12:32:05.833 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:05.833 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:05.833 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 10 monclient: dump: 2024-05-15T12:32:05.833 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:05.833 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 10 monclient: _finish_auth 0 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 10 monclient: _check_auth_tickets 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 1 -- 172.21.15.49:0/2948515942 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c039b50 con 0x7fedb013ffa0 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedb6f0b640 5 monclient: authenticate success, global_id 4207 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedb6f0b640 10 monclient: _renew_subs 2024-05-15T12:32:05.834 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedb6f0b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedb6f0b640 1 -- 172.21.15.49:0/2948515942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb00abdf0 con 0x7fedb013ffa0 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 1 -- 172.21.15.49:0/2948515942 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c035040 con 0x7fedb013ffa0 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedad7fa640 10 monclient: dump: 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:05.835 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:05.836 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.842+0000 7fedaeffd640 4 set_mon_vals no callback set 2024-05-15T12:32:05.836 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.843+0000 7fedb6f0b640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:05.836 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.843+0000 7fedb6f0b640 10 monclient: _renew_subs 2024-05-15T12:32:05.836 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.843+0000 7fedb6f0b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:05.836 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.843+0000 7fedb6f0b640 1 -- 172.21.15.49:0/2948515942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb01391f0 con 0x7fedb013ffa0 2024-05-15T12:32:05.836 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.843+0000 7fedb6f0b640 1 librados: init done 2024-05-15T12:32:05.836 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.844+0000 7fedb6f0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:05.836 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.844+0000 7fedb6f0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:05.836 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.844+0000 7fedb6f0b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:05.836 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.844+0000 7fedb6f0b640 1 -- 172.21.15.49:0/2948515942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb0068270 con 0x7fedb013ffa0 2024-05-15T12:32:05.862 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.873+0000 7fedad7fa640 1 -- 172.21.15.49:0/2948515942 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fed9c04b060 con 0x7fedb013ffa0 2024-05-15T12:32:05.863 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.874+0000 7fedad7fa640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fed9406ce20 0x7fed9406f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:05.863 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.874+0000 7fedb4ca7640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fed9406ce20 0x7fed9406f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:05.863 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.874+0000 7fedb4ca7640 10 monclient: get_auth_request con 0x7fed9406ce20 auth_method 0 2024-05-15T12:32:05.863 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.874+0000 7fedad7fa640 10 client.4207.objecter ms_handle_connect 0x7fed9406ce20 2024-05-15T12:32:05.863 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:05.874+0000 7fedb4ca7640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fed9406ce20 0x7fed9406f2e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7feda4000f60 tx=0x7feda4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:05.937 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:17.302 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:06.112 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.122+0000 7fedad7fa640 1 -- 172.21.15.49:0/2948515942 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fed9c0792c0 con 0x7fedb013ffa0 2024-05-15T12:32:06.112 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.122+0000 7fedad7fa640 10 client.4207.objecter ms_dispatch 0x7fedb0000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:32:06.112 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.122+0000 7fedad7fa640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:32:06.112 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.122+0000 7fedad7fa640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:32:06.112 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.123+0000 7fedad7fa640 20 client.4207.objecter dump_active .. 0 homeless 2024-05-15T12:32:06.112 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.123+0000 7fedad7fa640 1 -- 172.21.15.49:0/2948515942 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fed9c07a630 con 0x7fedb013ffa0 2024-05-15T12:32:06.112 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.123+0000 7fedad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:06.112 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.123+0000 7fedad7fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:06.228 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:17.566 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:06.285 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:17.628 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:06.444 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.455+0000 7fedb6f0b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:06.444 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.455+0000 7fedb6f0b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:06.444 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.455+0000 7fedb6f0b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:06.444 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.455+0000 7fedb6f0b640 1 -- 172.21.15.49:0/2948515942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedb00abfc0 con 0x7fedb013ffa0 2024-05-15T12:32:06.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.475+0000 7fedad7fa640 1 -- 172.21.15.49:0/2948515942 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fed9c07e6d0 con 0x7fedb013ffa0 2024-05-15T12:32:06.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.475+0000 7fedad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:06.464 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.475+0000 7fedad7fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:06.464 INFO:teuthology.misc.health.smithi049.stdout: 2024-05-15T12:32:06.464 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":10,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:31:48.140010+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:32:06.468 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 10 librados: watch_flush enter 2024-05-15T12:32:06.468 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 10 librados: watch_flush exit 2024-05-15T12:32:06.468 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 20 client.4207.objecter shutdown clearing up homeless session... 2024-05-15T12:32:06.468 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 10 client.4207.objecter successfully canceled tick 2024-05-15T12:32:06.468 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 1 -- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fed9406ce20 msgr2=0x7fed9406f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:06.468 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fed9406ce20 0x7fed9406f2e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7feda4000f60 tx=0x7feda4009040 comp rx=0 tx=0).stop 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 10 monclient: shutdown 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 1 -- 172.21.15.49:0/2948515942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedb013ffa0 msgr2=0x7fedb0135320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedb013ffa0 0x7fedb0135320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fed9c034040 tx=0x7fed9c0046c0 comp rx=0 tx=0).stop 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 1 -- 172.21.15.49:0/2948515942 shutdown_connections 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedb0142be0 0x7fedb0135860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fed9406ce20 0x7fed9406f2e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedb013ffa0 0x7fedb0135320 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 1 --2- 172.21.15.49:0/2948515942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedb00b23b0 0x7fedb013cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.478+0000 7fedb6f0b640 1 -- 172.21.15.49:0/2948515942 >> 172.21.15.49:0/2948515942 conn(0x7fedb00b8390 msgr2=0x7fedb00b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.479+0000 7fedb6f0b640 1 -- 172.21.15.49:0/2948515942 shutdown_connections 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.479+0000 7fedb6f0b640 1 -- 172.21.15.49:0/2948515942 wait complete. 2024-05-15T12:32:06.469 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:06.479+0000 7fedb6f0b640 1 librados: shutdown 2024-05-15T12:32:06.479 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-15T12:32:06.620 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:17.974 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:06.959 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:18.320 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:07.203 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:18.552 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:10.914 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:22.279 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:10.919 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:22.283 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:11.099 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:22.461 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:11.104 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:22.466 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:11.420 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:22.784 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:11.425 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:22.790 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:11.667 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:23.010 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:11.671 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:23.014 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:11.971 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:23.309 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:11.975 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:23.314 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:12.792 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:24.152 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:12.796 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:24.157 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:12.843 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:24.197 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:12.847 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:24.201 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:12.965 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:24.314 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:12.969 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:24.318 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:13.285 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.424+0000 7fc8725fe640 10 monclient: get_monmap_and_config 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.424+0000 7fc8725fe640 10 monclient: build_initial_monmap 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.425+0000 7fc8725fe640 10 monclient: monmap: 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:32:13.425535+0000 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:32:13.425535+0000 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:13.414 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:13.415 INFO:teuthology.misc.health.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:13.415 INFO:teuthology.misc.health.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:13.415 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:13.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.426+0000 7fc8725fe640 1 Processor -- start 2024-05-15T12:32:13.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.426+0000 7fc8725fe640 1 -- start start 2024-05-15T12:32:13.415 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.426+0000 7fc8725fe640 10 monclient: init 2024-05-15T12:32:13.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:13.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-05-15T12:32:13.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc86c13ffa0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:13.416 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc86c0b23b0 0x7fc86c1426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:13.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc86c0b23b0 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:13.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc86c142be0 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:13.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c0ab160 con 0x7fc86c13ffa0 2024-05-15T12:32:13.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c0ab440 con 0x7fc86c142be0 2024-05-15T12:32:13.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:13.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86c143000 con 0x7fc86c0b23b0 2024-05-15T12:32:13.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:13.417 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.427+0000 7fc8725fe640 10 monclient(hunting): authenticate will time out at 1491.588972s 2024-05-15T12:32:13.470 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.481+0000 7fc870b9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc86c142be0 0x7fc86c14cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:38658/0 (socket says 172.21.15.49:38658) 2024-05-15T12:32:13.471 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.481+0000 7fc870b9b640 1 -- 172.21.15.49:0/3182484604 learned_addr learned my addr 172.21.15.49:0/3182484604 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:13.478 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.489+0000 7fc86b7fe640 1 --2- 172.21.15.49:0/3182484604 >> v2:172.21.15.49:3300/0 conn(0x7fc86c0b23b0 0x7fc86c1426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:13.479 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.491+0000 7fc86b7fe640 10 monclient(hunting): get_auth_request con 0x7fc86c0b23b0 auth_method 0 2024-05-15T12:32:13.480 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.491+0000 7fc86b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:13.480 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.491+0000 7fc86b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:13.480 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.491+0000 7fc86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:13.481 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.492+0000 7fc86b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:13.481 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.492+0000 7fc86b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:13.481 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.492+0000 7fc86b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:13.486 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.496+0000 7fc86b7fe640 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2024-05-15T12:32:13.486 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.496+0000 7fc86b7fe640 1 -- 172.21.15.49:0/3182484604 >> v1:172.21.15.88:6789/0 conn(0x7fc86c142be0 legacy=0x7fc86c14cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:13.486 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.496+0000 7fc86b7fe640 1 -- 172.21.15.49:0/3182484604 >> v1:172.21.15.49:6789/0 conn(0x7fc86c13ffa0 legacy=0x7fc86c0728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:13.486 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.496+0000 7fc86b7fe640 10 monclient: _finish_hunting 0 2024-05-15T12:32:13.486 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.496+0000 7fc86b7fe640 1 monclient: found mon.noname-a 2024-05-15T12:32:13.486 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.496+0000 7fc86b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:13.486 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.496+0000 7fc86b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:13.486 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.496+0000 7fc86b7fe640 1 -- 172.21.15.49:0/3182484604 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0a9ae0 con 0x7fc86c0b23b0 2024-05-15T12:32:13.489 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.500+0000 7fc86b7fe640 1 --2- 172.21.15.49:0/3182484604 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc86c0b23b0 0x7fc86c1426a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8600097a0 tx=0x7fc86002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45db4b43e8e8dbe9 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:13.504 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 1 -- 172.21.15.49:0/3182484604 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8600315a0 con 0x7fc86c0b23b0 2024-05-15T12:32:13.504 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 10 monclient: dump: 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 10 monclient: _finish_auth 0 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:13.505 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 1 -- 172.21.15.49:0/3182484604 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc860037070 con 0x7fc86c0b23b0 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 1 -- 172.21.15.49:0/3182484604 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc860031960 con 0x7fc86c0b23b0 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc86affd640 10 monclient: dump: 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:13.506 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:13.507 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:13.507 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:13.507 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc8725fe640 5 monclient: authenticate success, global_id 4209 2024-05-15T12:32:13.507 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc8725fe640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:13.507 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.515+0000 7fc8725fe640 4 set_mon_vals no callback set 2024-05-15T12:32:13.511 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.516+0000 7fc8725fe640 10 monclient: shutdown 2024-05-15T12:32:13.511 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.516+0000 7fc8725fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:13.511 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.516+0000 7fc8725fe640 1 -- 172.21.15.49:0/3182484604 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc86c0b23b0 msgr2=0x7fc86c1426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:13.511 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.516+0000 7fc8725fe640 1 --2- 172.21.15.49:0/3182484604 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc86c0b23b0 0x7fc86c1426a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8600097a0 tx=0x7fc86002b840 comp rx=0 tx=0).stop 2024-05-15T12:32:13.511 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.516+0000 7fc8725fe640 1 -- 172.21.15.49:0/3182484604 shutdown_connections 2024-05-15T12:32:13.511 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.516+0000 7fc8725fe640 1 --2- 172.21.15.49:0/3182484604 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc86c0b23b0 0x7fc86c1426a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:13.511 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.516+0000 7fc8725fe640 1 -- 172.21.15.49:0/3182484604 >> 172.21.15.49:0/3182484604 conn(0x7fc86c0b8390 msgr2=0x7fc86c0b87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:13.511 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.516+0000 7fc8725fe640 1 -- 172.21.15.49:0/3182484604 shutdown_connections 2024-05-15T12:32:13.511 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.516+0000 7fc8725fe640 1 -- 172.21.15.49:0/3182484604 wait complete. 2024-05-15T12:32:13.511 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.517+0000 7fc8725fe640 10 monclient: build_initial_monmap 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.517+0000 7fc8725fe640 10 monclient: monmap: 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:created 0.000000 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.518+0000 7fc8725fe640 1 librados: starting msgr at 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.518+0000 7fc8725fe640 1 librados: starting objecter 2024-05-15T12:32:13.512 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.518+0000 7fc8725fe640 1 Processor -- start 2024-05-15T12:32:13.513 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.518+0000 7fc8725fe640 1 -- start start 2024-05-15T12:32:13.513 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.518+0000 7fc8725fe640 1 librados: setting wanted keys 2024-05-15T12:32:13.513 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.518+0000 7fc8725fe640 1 librados: calling monclient init 2024-05-15T12:32:13.513 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.518+0000 7fc8725fe640 10 monclient: init 2024-05-15T12:32:13.513 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.519+0000 7fc8725fe640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:13.513 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.519+0000 7fc8725fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:32:13.513 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.519+0000 7fc8725fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc86c0b23b0 0x7fc86c135320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:13.513 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.519+0000 7fc8725fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc86c0b23b0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:13.513 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.519+0000 7fc8725fe640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc86c13ffa0 0x7fc86c13a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:13.514 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc86c13ffa0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:13.514 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c142be0 0x7fc86c13aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:13.514 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 10 monclient(hunting): picked mon.noname-b con 0x7fc86c142be0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:13.514 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:13.514 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c14e7d0 con 0x7fc86c0b23b0 2024-05-15T12:32:13.514 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:13.515 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c14e650 con 0x7fc86c142be0 2024-05-15T12:32:13.515 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:13.515 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c14e950 con 0x7fc86c13ffa0 2024-05-15T12:32:13.515 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:13.515 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc8725fe640 10 monclient(hunting): authenticate will time out at 1491.682308s 2024-05-15T12:32:13.515 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.520+0000 7fc870b9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c142be0 0x7fc86c13aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:13.515 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.521+0000 7fc870b9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c142be0 0x7fc86c13aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:55548/0 (socket says 172.21.15.49:55548) 2024-05-15T12:32:13.515 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.521+0000 7fc870b9b640 1 -- 172.21.15.49:0/589471778 learned_addr learned my addr 172.21.15.49:0/589471778 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:13.515 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.521+0000 7fc870b9b640 10 monclient(hunting): get_auth_request con 0x7fc86c142be0 auth_method 0 2024-05-15T12:32:13.515 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.521+0000 7fc870b9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:13.516 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.521+0000 7fc870b9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:13.516 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.521+0000 7fc870b9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:13.516 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.521+0000 7fc870b9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:13.516 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.521+0000 7fc870b9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:13.516 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.521+0000 7fc870b9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:13.516 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.521+0000 7fc86b7fe640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc86c13ffa0 0x7fc86c13a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:13.516 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.522+0000 7fc86b7fe640 10 monclient(hunting): get_auth_request con 0x7fc86c13ffa0 auth_method 0 2024-05-15T12:32:13.516 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.522+0000 7fc86b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:13.516 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.522+0000 7fc86b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:13.516 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.522+0000 7fc86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:13.517 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.522+0000 7fc86b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:13.517 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.522+0000 7fc86b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:13.517 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.522+0000 7fc86b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:13.517 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.524+0000 7fc870b9b640 10 monclient(hunting): handle_auth_done global_id 4213 payload 824 2024-05-15T12:32:13.517 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.524+0000 7fc870b9b640 1 -- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc86c13ffa0 msgr2=0x7fc86c13a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:13.517 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.524+0000 7fc870b9b640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc86c13ffa0 0x7fc86c13a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:13.517 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.524+0000 7fc870b9b640 1 -- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc86c0b23b0 msgr2=0x7fc86c135320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:13.518 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.524+0000 7fc870b9b640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc86c0b23b0 0x7fc86c135320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:13.518 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.524+0000 7fc870b9b640 10 monclient: _finish_hunting 0 2024-05-15T12:32:13.518 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.524+0000 7fc870b9b640 1 monclient: found mon.noname-b 2024-05-15T12:32:13.518 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.524+0000 7fc870b9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:13.518 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.524+0000 7fc870b9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:13.518 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.524+0000 7fc870b9b640 1 -- 172.21.15.49:0/589471778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c136150 con 0x7fc86c142be0 2024-05-15T12:32:13.518 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.527+0000 7fc870b9b640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c142be0 0x7fc86c13aa30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc86400caa0 tx=0x7fc86400cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:13.518 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.527+0000 7fc868ff9640 10 client.?.objecter ms_handle_connect 0x7fc86c142be0 2024-05-15T12:32:13.518 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.527+0000 7fc868ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:13.522 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.530+0000 7fc868ff9640 1 -- 172.21.15.49:0/589471778 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864017070 con 0x7fc86c142be0 2024-05-15T12:32:13.522 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.530+0000 7fc868ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:13.522 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.530+0000 7fc868ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:13.522 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.530+0000 7fc868ff9640 10 monclient: dump: 2024-05-15T12:32:13.522 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:13.522 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:13.522 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:13.522 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:13.522 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:13.522 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc868ff9640 10 monclient: _finish_auth 0 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc868ff9640 10 monclient: _check_auth_tickets 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc868ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc868ff9640 1 -- 172.21.15.49:0/589471778 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86400ac00 con 0x7fc86c142be0 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc868ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc868ff9640 1 -- 172.21.15.49:0/589471778 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864012620 con 0x7fc86c142be0 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc868ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc868ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc868ff9640 10 monclient: dump: 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:13.523 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc8725fe640 5 monclient: authenticate success, global_id 4213 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc8725fe640 10 monclient: _renew_subs 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc8725fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc8725fe640 1 -- 172.21.15.49:0/589471778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c136430 con 0x7fc86c142be0 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc86a7fc640 4 set_mon_vals no callback set 2024-05-15T12:32:13.524 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc8725fe640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:13.525 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc8725fe640 10 monclient: _renew_subs 2024-05-15T12:32:13.525 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc8725fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:13.525 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc8725fe640 1 -- 172.21.15.49:0/589471778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c13d1f0 con 0x7fc86c142be0 2024-05-15T12:32:13.525 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.531+0000 7fc8725fe640 1 librados: init done 2024-05-15T12:32:13.525 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.532+0000 7fc8725fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:13.525 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.532+0000 7fc8725fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:13.525 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.532+0000 7fc8725fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:13.525 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.532+0000 7fc8725fe640 1 -- 172.21.15.49:0/589471778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c134c80 con 0x7fc86c142be0 2024-05-15T12:32:13.539 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.550+0000 7fc868ff9640 1 -- 172.21.15.49:0/589471778 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fc86401a070 con 0x7fc86c142be0 2024-05-15T12:32:13.540 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.551+0000 7fc868ff9640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc85806cdd0 0x7fc85806f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:13.540 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.551+0000 7fc86bfff640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc85806cdd0 0x7fc85806f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:13.540 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.551+0000 7fc86bfff640 10 monclient: get_auth_request con 0x7fc85806cdd0 auth_method 0 2024-05-15T12:32:13.540 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.551+0000 7fc868ff9640 10 client.4213.objecter ms_handle_connect 0x7fc85806cdd0 2024-05-15T12:32:13.540 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.551+0000 7fc86bfff640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc85806cdd0 0x7fc85806f290 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc85c001560 tx=0x7fc85c007990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:13.542 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.553+0000 7fc868ff9640 1 -- 172.21.15.49:0/589471778 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc8640577c0 con 0x7fc86c142be0 2024-05-15T12:32:13.542 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.553+0000 7fc868ff9640 10 client.4213.objecter ms_dispatch 0x7fc86c000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:32:13.542 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.553+0000 7fc868ff9640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:32:13.542 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.553+0000 7fc868ff9640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:32:13.542 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.553+0000 7fc868ff9640 20 client.4213.objecter dump_active .. 0 homeless 2024-05-15T12:32:13.920 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.929+0000 7fc868ff9640 1 -- 172.21.15.49:0/589471778 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc864021020 con 0x7fc86c142be0 2024-05-15T12:32:13.920 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.929+0000 7fc868ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:13.920 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:13.929+0000 7fc868ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:14.247 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.257+0000 7fc8725fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:14.247 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.257+0000 7fc8725fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:14.247 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.257+0000 7fc8725fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:14.247 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.257+0000 7fc8725fe640 1 -- 172.21.15.49:0/589471778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc86c0aa3f0 con 0x7fc86c142be0 2024-05-15T12:32:14.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.272+0000 7fc868ff9640 1 -- 172.21.15.49:0/589471778 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fc864058b30 con 0x7fc86c142be0 2024-05-15T12:32:14.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.272+0000 7fc868ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:14.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.272+0000 7fc868ff9640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:14.263 INFO:teuthology.misc.health.smithi049.stdout: 2024-05-15T12:32:14.263 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":10,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:31:48.140010+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 10 librados: watch_flush enter 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 10 librados: watch_flush exit 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 20 client.4213.objecter shutdown clearing up homeless session... 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 10 client.4213.objecter successfully canceled tick 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 -- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc85806cdd0 msgr2=0x7fc85806f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc85806cdd0 0x7fc85806f290 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc85c001560 tx=0x7fc85c007990 comp rx=0 tx=0).stop 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 10 monclient: shutdown 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 -- 172.21.15.49:0/589471778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c142be0 msgr2=0x7fc86c13aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c142be0 0x7fc86c13aa30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc86400caa0 tx=0x7fc86400cf60 comp rx=0 tx=0).stop 2024-05-15T12:32:14.266 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 -- 172.21.15.49:0/589471778 shutdown_connections 2024-05-15T12:32:14.267 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc85806cdd0 0x7fc85806f290 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:14.267 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c142be0 0x7fc86c13aa30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:14.267 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc86c13ffa0 0x7fc86c13a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:14.267 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 --2- 172.21.15.49:0/589471778 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc86c0b23b0 0x7fc86c135320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:14.267 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 -- 172.21.15.49:0/589471778 >> 172.21.15.49:0/589471778 conn(0x7fc86c0b8390 msgr2=0x7fc86c0b7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:14.267 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 -- 172.21.15.49:0/589471778 shutdown_connections 2024-05-15T12:32:14.267 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 -- 172.21.15.49:0/589471778 wait complete. 2024-05-15T12:32:14.267 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:14.275+0000 7fc8725fe640 1 librados: shutdown 2024-05-15T12:32:14.277 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-15T12:32:15.405 INFO:tasks.ceph.osd.6.smithi088.stderr:==00:00:00:26.767 34956== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:15.409 INFO:tasks.ceph.osd.4.smithi088.stderr:==00:00:00:26.774 34954== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:15.940 INFO:tasks.ceph.osd.5.smithi088.stderr:==00:00:00:27.305 34955== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:16.302 INFO:tasks.ceph.osd.3.smithi049.stderr:==00:00:00:27.641 34600== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:16.453 INFO:tasks.ceph.osd.7.smithi088.stderr:==00:00:00:27.795 34957== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:17.099 INFO:tasks.ceph.osd.0.smithi049.stderr:==00:00:00:28.459 34597== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:17.402 INFO:tasks.ceph.osd.2.smithi049.stderr:==00:00:00:28.756 34598== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:17.440 INFO:tasks.ceph.osd.1.smithi049.stderr:==00:00:00:28.790 34599== Warning: unimplemented fcntl command: 1038 2024-05-15T12:32:21.084 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:32:21.226 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.235+0000 7f37a2abe640 10 monclient: get_monmap_and_config 2024-05-15T12:32:21.226 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.235+0000 7f37a2abe640 10 monclient: build_initial_monmap 2024-05-15T12:32:21.226 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.235+0000 7f37a2abe640 10 monclient: monmap: 2024-05-15T12:32:21.226 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:21.226 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:21.226 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:32:21.237247+0000 2024-05-15T12:32:21.226 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:32:21.237247+0000 2024-05-15T12:32:21.226 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:21.226 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:21.226 INFO:teuthology.misc.health.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:21.227 INFO:teuthology.misc.health.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:21.227 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:21.227 INFO:teuthology.misc.health.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:21.227 INFO:teuthology.misc.health.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:21.227 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:21.230 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.236+0000 7f37a2abe640 1 Processor -- start 2024-05-15T12:32:21.230 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.236+0000 7f37a2abe640 1 -- start start 2024-05-15T12:32:21.230 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.236+0000 7f37a2abe640 10 monclient: init 2024-05-15T12:32:21.230 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.237+0000 7f37a2abe640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:21.230 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.237+0000 7f37a2abe640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-05-15T12:32:21.230 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.237+0000 7f37a2abe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f379c13ffa0 0x7f379c0728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:21.230 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.237+0000 7f37a2abe640 10 monclient(hunting): picked mon.noname-c con 0x7f379c13ffa0 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a2abe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f379c0b23b0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a2abe640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c14af20 0x7f379c14d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a2abe640 10 monclient(hunting): picked mon.noname-b con 0x7f379c14af20 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a2abe640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0ab160 con 0x7f379c0b23b0 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a2abe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a2abe640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f379c0ab440 con 0x7f379c13ffa0 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a2abe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a2abe640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0a9ae0 con 0x7f379c14af20 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a2abe640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a2abe640 10 monclient(hunting): authenticate will time out at 1499.401189s 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.238+0000 7f37a085a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f379c13ffa0 0x7f379c0728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:21.231 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f379bfff640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f379c0b23b0 0x7f379c1426a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:51106/0 (socket says 172.21.15.49:51106) 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f379bfff640 1 -- 172.21.15.49:0/2847912053 learned_addr learned my addr 172.21.15.49:0/2847912053 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a105b640 1 --2- 172.21.15.49:0/2847912053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c14af20 0x7f379c14d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a085a640 10 monclient(hunting): get_auth_request con 0x7f379c13ffa0 auth_method 0 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a085a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a085a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a085a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a105b640 10 monclient(hunting): get_auth_request con 0x7f379c14af20 auth_method 0 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a105b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a105b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a105b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a085a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a085a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.239+0000 7f37a085a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:21.232 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.240+0000 7f37a105b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:21.233 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.240+0000 7f37a105b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:21.233 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.240+0000 7f37a105b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:21.234 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.242+0000 7f37a085a640 10 monclient(hunting): handle_auth_done global_id 4216 payload 274 2024-05-15T12:32:21.234 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.242+0000 7f37a085a640 1 -- 172.21.15.49:0/2847912053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c14af20 msgr2=0x7f379c14d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:21.234 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.242+0000 7f37a085a640 1 --2- 172.21.15.49:0/2847912053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c14af20 0x7f379c14d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:21.234 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.242+0000 7f37a085a640 1 -- 172.21.15.49:0/2847912053 >> v1:172.21.15.49:6789/0 conn(0x7f379c0b23b0 legacy=0x7f379c1426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:21.234 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.242+0000 7f37a085a640 10 monclient: _finish_hunting 0 2024-05-15T12:32:21.234 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.242+0000 7f37a085a640 1 monclient: found mon.noname-c 2024-05-15T12:32:21.234 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.242+0000 7f37a085a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:21.234 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.242+0000 7f37a085a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:21.234 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.242+0000 7f37a085a640 1 -- 172.21.15.49:0/2847912053 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c072df0 con 0x7f379c13ffa0 2024-05-15T12:32:21.235 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.245+0000 7f37a085a640 1 --2- 172.21.15.49:0/2847912053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c13ffa0 0x7f379c0728b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f37900097a0 tx=0x7f379002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2279f70c60eb431 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:21.242 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.250+0000 7f379b7fe640 1 -- 172.21.15.49:0/2847912053 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790031780 con 0x7f379c13ffa0 2024-05-15T12:32:21.242 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.250+0000 7f379b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.250+0000 7f379b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f379b7fe640 10 monclient: dump: 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:21.243 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f379b7fe640 10 monclient: _finish_auth 0 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f379b7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f379b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f37a2abe640 5 monclient: authenticate success, global_id 4216 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f37a2abe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f379b7fe640 1 -- 172.21.15.49:0/2847912053 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3790037070 con 0x7f379c13ffa0 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f379b7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f37a2abe640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f37a2abe640 4 set_mon_vals no callback set 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f379b7fe640 1 -- 172.21.15.49:0/2847912053 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790031b40 con 0x7f379c13ffa0 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f379b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f379b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.251+0000 7f379b7fe640 10 monclient: dump: 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:21.244 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.252+0000 7f37a2abe640 10 monclient: shutdown 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.252+0000 7f37a2abe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.252+0000 7f37a2abe640 1 -- 172.21.15.49:0/2847912053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c13ffa0 msgr2=0x7f379c0728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.252+0000 7f37a2abe640 1 --2- 172.21.15.49:0/2847912053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c13ffa0 0x7f379c0728b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f37900097a0 tx=0x7f379002b820 comp rx=0 tx=0).stop 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.252+0000 7f37a2abe640 1 -- 172.21.15.49:0/2847912053 shutdown_connections 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.252+0000 7f37a2abe640 1 --2- 172.21.15.49:0/2847912053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c14af20 0x7f379c14d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:21.245 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.252+0000 7f37a2abe640 1 --2- 172.21.15.49:0/2847912053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c13ffa0 0x7f379c0728b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:21.246 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.252+0000 7f37a2abe640 1 -- 172.21.15.49:0/2847912053 >> 172.21.15.49:0/2847912053 conn(0x7f379c0b8390 msgr2=0x7f379c0b87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:21.246 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.252+0000 7f37a2abe640 1 -- 172.21.15.49:0/2847912053 shutdown_connections 2024-05-15T12:32:21.246 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.252+0000 7f37a2abe640 1 -- 172.21.15.49:0/2847912053 wait complete. 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.253+0000 7f37a2abe640 10 monclient: build_initial_monmap 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.253+0000 7f37a2abe640 10 monclient: monmap: 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:created 0.000000 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:21.250 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.253+0000 7f37a2abe640 1 librados: starting msgr at 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.253+0000 7f37a2abe640 1 librados: starting objecter 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.254+0000 7f37a2abe640 1 Processor -- start 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.254+0000 7f37a2abe640 1 -- start start 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.254+0000 7f37a2abe640 1 librados: setting wanted keys 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.254+0000 7f37a2abe640 1 librados: calling monclient init 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.254+0000 7f37a2abe640 10 monclient: init 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c0b23b0 0x7f379c0ad340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 10 monclient(hunting): picked mon.noname-c con 0x7f379c0b23b0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:21.251 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f379c13ffa0 0x7f379c0aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 10 monclient(hunting): picked mon.noname-a con 0x7f379c13ffa0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c14af20 0x7f379c0b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 10 monclient(hunting): picked mon.noname-b con 0x7f379c14af20 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f379bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f379c13ffa0 0x7f379c0aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c142db0 con 0x7f379c13ffa0 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0aa210 con 0x7f379c14af20 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.255+0000 7f37a2abe640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c142f30 con 0x7f379c0b23b0 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f37a2abe640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f37a2abe640 10 monclient(hunting): authenticate will time out at 1499.418467s 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f37a085a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c0b23b0 0x7f379c0ad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f379bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f379c13ffa0 0x7f379c0aff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45144/0 (socket says 172.21.15.49:45144) 2024-05-15T12:32:21.252 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f379bfff640 1 -- 172.21.15.49:0/2167218253 learned_addr learned my addr 172.21.15.49:0/2167218253 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f379bfff640 10 monclient(hunting): get_auth_request con 0x7f379c13ffa0 auth_method 0 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f379bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f379bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f379bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f37a085a640 10 monclient(hunting): get_auth_request con 0x7f379c0b23b0 auth_method 0 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f37a085a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f37a085a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f37a085a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.256+0000 7f37a105b640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c14af20 0x7f379c0b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f37a105b640 10 monclient(hunting): get_auth_request con 0x7f379c14af20 auth_method 0 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f37a105b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f37a105b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f37a105b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f379bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:21.253 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f379bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:21.254 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f379bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:21.254 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f37a085a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:21.254 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f37a085a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:21.254 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f37a085a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:21.254 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f37a105b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:21.254 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f37a105b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:21.254 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.257+0000 7f37a105b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:21.254 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.261+0000 7f37a085a640 10 monclient(hunting): handle_auth_done global_id 4178 payload 824 2024-05-15T12:32:21.254 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.261+0000 7f37a085a640 1 -- 172.21.15.49:0/2167218253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c14af20 msgr2=0x7f379c0b0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:21.255 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.261+0000 7f37a085a640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c14af20 0x7f379c0b0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:21.255 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.261+0000 7f37a085a640 1 -- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f379c13ffa0 msgr2=0x7f379c0aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:21.255 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.261+0000 7f37a085a640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f379c13ffa0 0x7f379c0aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:21.255 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.261+0000 7f37a085a640 10 monclient: _finish_hunting 0 2024-05-15T12:32:21.255 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.261+0000 7f37a085a640 1 monclient: found mon.noname-c 2024-05-15T12:32:21.255 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.261+0000 7f37a085a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:21.255 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.261+0000 7f37a085a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:21.255 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.261+0000 7f37a085a640 1 -- 172.21.15.49:0/2167218253 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c143770 con 0x7f379c0b23b0 2024-05-15T12:32:21.255 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.265+0000 7f37a085a640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c0b23b0 0x7f379c0ad340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3790033000 tx=0x7f37900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:21.256 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.265+0000 7f37997fa640 10 client.?.objecter ms_handle_connect 0x7f379c0b23b0 2024-05-15T12:32:21.256 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.265+0000 7f37997fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:21.258 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37997fa640 1 -- 172.21.15.49:0/2167218253 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790032820 con 0x7f379c0b23b0 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37997fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37997fa640 10 monclient: dump: 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:21.259 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37997fa640 10 monclient: _finish_auth 0 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37997fa640 10 monclient: _check_auth_tickets 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37a2abe640 5 monclient: authenticate success, global_id 4178 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37a2abe640 10 monclient: _renew_subs 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37a2abe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37a2abe640 1 -- 172.21.15.49:0/2167218253 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c143a50 con 0x7f379c0b23b0 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37a2abe640 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37a2abe640 10 monclient: _renew_subs 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37a2abe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37a2abe640 1 -- 172.21.15.49:0/2167218253 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379c0b0440 con 0x7f379c0b23b0 2024-05-15T12:32:21.260 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.268+0000 7f37a2abe640 1 librados: init done 2024-05-15T12:32:21.261 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.269+0000 7f37a2abe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:21.261 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.269+0000 7f37a2abe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.270+0000 7f37a2abe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.270+0000 7f37a2abe640 1 -- 172.21.15.49:0/2167218253 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379c139f60 con 0x7f379c0b23b0 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.270+0000 7f37997fa640 1 -- 172.21.15.49:0/2167218253 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3790037020 con 0x7f379c0b23b0 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.271+0000 7f37997fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.271+0000 7f37997fa640 1 -- 172.21.15.49:0/2167218253 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790032be0 con 0x7f379c0b23b0 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.271+0000 7f37997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.271+0000 7f37997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.271+0000 7f37997fa640 10 monclient: dump: 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:21.262 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:21.263 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:21.263 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:21.263 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:21.263 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:21.263 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.271+0000 7f379affd640 4 set_mon_vals no callback set 2024-05-15T12:32:21.587 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.597+0000 7f37997fa640 1 -- 172.21.15.49:0/2167218253 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f379004e440 con 0x7f379c0b23b0 2024-05-15T12:32:21.588 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.598+0000 7f37997fa640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f378006cdd0 0x7f378006f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:21.588 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.598+0000 7f379bfff640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f378006cdd0 0x7f378006f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:21.588 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.598+0000 7f379bfff640 10 monclient: get_auth_request con 0x7f378006cdd0 auth_method 0 2024-05-15T12:32:21.589 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.598+0000 7f379bfff640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f378006cdd0 0x7f378006f290 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f379c13f440 tx=0x7f3788009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:21.589 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.598+0000 7f37997fa640 10 client.4178.objecter ms_handle_connect 0x7f378006cdd0 2024-05-15T12:32:21.614 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.624+0000 7f37997fa640 1 -- 172.21.15.49:0/2167218253 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f37900aa580 con 0x7f379c0b23b0 2024-05-15T12:32:21.615 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.624+0000 7f37997fa640 10 client.4178.objecter ms_dispatch 0x7f379c000b90 osd_map(10..10 src has 1..10) v4 2024-05-15T12:32:21.615 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.624+0000 7f37997fa640 3 client.4178.objecter handle_osd_map got epochs [10,10] > 0 2024-05-15T12:32:21.615 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.624+0000 7f37997fa640 3 client.4178.objecter handle_osd_map decoding full epoch 10 2024-05-15T12:32:21.615 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.624+0000 7f37997fa640 20 client.4178.objecter dump_active .. 0 homeless 2024-05-15T12:32:21.658 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.668+0000 7f37997fa640 1 -- 172.21.15.49:0/2167218253 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3790040070 con 0x7f379c0b23b0 2024-05-15T12:32:21.658 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.668+0000 7f37997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:21.658 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:21.668+0000 7f37997fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:22.046 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.055+0000 7f37a2abe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:22.046 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.055+0000 7f37a2abe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:22.046 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.055+0000 7f37a2abe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:22.046 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.055+0000 7f37a2abe640 1 -- 172.21.15.49:0/2167218253 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f379c0aa3f0 con 0x7f379c0b23b0 2024-05-15T12:32:22.134 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.144+0000 7f37997fa640 1 -- 172.21.15.49:0/2167218253 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f379002f050 con 0x7f379c0b23b0 2024-05-15T12:32:22.134 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.144+0000 7f37997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:22.134 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.144+0000 7f37997fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:22.134 INFO:teuthology.misc.health.smithi049.stdout: 2024-05-15T12:32:22.135 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":10,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:31:48.140010+0000","last_up_change":"0.000000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:32:22.138 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 10 librados: watch_flush enter 2024-05-15T12:32:22.138 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 10 librados: watch_flush exit 2024-05-15T12:32:22.138 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 20 client.4178.objecter shutdown clearing up homeless session... 2024-05-15T12:32:22.138 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 10 client.4178.objecter successfully canceled tick 2024-05-15T12:32:22.138 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 1 -- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f378006cdd0 msgr2=0x7f378006f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:22.138 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f378006cdd0 0x7f378006f290 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f379c13f440 tx=0x7f3788009040 comp rx=0 tx=0).stop 2024-05-15T12:32:22.139 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 10 monclient: shutdown 2024-05-15T12:32:22.139 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:22.139 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 1 -- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c0b23b0 msgr2=0x7f379c0ad340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:22.139 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c0b23b0 0x7f379c0ad340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3790033000 tx=0x7f37900046c0 comp rx=0 tx=0).stop 2024-05-15T12:32:22.139 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 1 -- 172.21.15.49:0/2167218253 shutdown_connections 2024-05-15T12:32:22.139 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f378006cdd0 0x7f378006f290 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:22.139 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c14af20 0x7f379c0b0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:22.139 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f379c13ffa0 0x7f379c0aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:22.139 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 1 --2- 172.21.15.49:0/2167218253 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f379c0b23b0 0x7f379c0ad340 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:22.139 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.147+0000 7f37a2abe640 1 -- 172.21.15.49:0/2167218253 >> 172.21.15.49:0/2167218253 conn(0x7f379c0b8390 msgr2=0x7f379c0b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:22.140 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.149+0000 7f37a2abe640 1 -- 172.21.15.49:0/2167218253 shutdown_connections 2024-05-15T12:32:22.140 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.150+0000 7f37a2abe640 1 -- 172.21.15.49:0/2167218253 wait complete. 2024-05-15T12:32:22.140 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:22.150+0000 7f37a2abe640 1 librados: shutdown 2024-05-15T12:32:22.155 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-15T12:32:23.856 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-15T12:32:23.863+0000 5dce500 -1 osd.4 0 log_to_monitors true 2024-05-15T12:32:24.067 INFO:tasks.ceph.osd.6.smithi088.stderr:2024-05-15T12:32:24.074+0000 5dce500 -1 osd.6 0 log_to_monitors true 2024-05-15T12:32:24.286 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:24.293+0000 5dce500 -1 osd.5 0 log_to_monitors true 2024-05-15T12:32:25.053 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:25.060+0000 5dce500 -1 osd.7 0 log_to_monitors true 2024-05-15T12:32:25.311 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:32:25.318+0000 5dce500 -1 osd.3 0 log_to_monitors true 2024-05-15T12:32:26.341 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:32:26.348+0000 5dce500 -1 osd.2 0 log_to_monitors true 2024-05-15T12:32:26.674 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:32:26.680+0000 5dce500 -1 osd.0 0 log_to_monitors true 2024-05-15T12:32:26.931 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-15T12:32:26.937+0000 c49a640 -1 osd.4 0 waiting for initial osdmap 2024-05-15T12:32:26.978 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:26.951+0000 5dce500 -1 osd.1 0 log_to_monitors true 2024-05-15T12:32:26.978 INFO:tasks.ceph.osd.6.smithi088.stderr:2024-05-15T12:32:26.963+0000 c49a640 -1 osd.6 0 waiting for initial osdmap 2024-05-15T12:32:27.344 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-15T12:32:27.340+0000 10260640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:32:27.407 INFO:tasks.ceph.osd.6.smithi088.stderr:2024-05-15T12:32:27.403+0000 10260640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:32:28.962 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.081+0000 7f1277b1b640 10 monclient: get_monmap_and_config 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.081+0000 7f1277b1b640 10 monclient: build_initial_monmap 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.081+0000 7f1277b1b640 10 monclient: monmap: 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:32:29.083428+0000 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:32:29.083428+0000 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:29.072 INFO:teuthology.misc.health.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:29.073 INFO:teuthology.misc.health.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:29.073 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:29.074 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.082+0000 7f1277b1b640 1 Processor -- start 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.082+0000 7f1277b1b640 1 -- start start 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.082+0000 7f1277b1b640 10 monclient: init 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f127013ffa0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f12700b23b0 0x7f12701426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 10 monclient(hunting): picked mon.noname-c con 0x7f12700b23b0 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127014cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 10 monclient(hunting): picked mon.noname-b con 0x7f1270142be0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12700ab160 con 0x7f127013ffa0 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1270143000 con 0x7f12700b23b0 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:29.075 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.083+0000 7f1277b1b640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12700ab440 con 0x7f1270142be0 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.084+0000 7f1277b1b640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.084+0000 7f1277b1b640 10 monclient(hunting): authenticate will time out at 1507.246445s 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.084+0000 7f12750b6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f12700b23b0 0x7f12701426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.084+0000 7f12760b8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127014cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.084+0000 7f12758b7640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f127013ffa0 0x7f12700728b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34310/0 (socket says 172.21.15.49:34310) 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.084+0000 7f12758b7640 1 -- 172.21.15.49:0/3631365521 learned_addr learned my addr 172.21.15.49:0/3631365521 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12760b8640 10 monclient(hunting): get_auth_request con 0x7f1270142be0 auth_method 0 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12760b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12760b8640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12760b8640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12750b6640 10 monclient(hunting): get_auth_request con 0x7f12700b23b0 auth_method 0 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12750b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12750b6640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:29.076 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12750b6640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:29.077 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12760b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:29.077 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12760b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:29.077 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12760b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:29.077 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12750b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:29.077 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12750b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:29.078 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.085+0000 7f12750b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:29.078 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.087+0000 7f12760b8640 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2024-05-15T12:32:29.078 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.087+0000 7f12760b8640 1 -- 172.21.15.49:0/3631365521 >> v2:172.21.15.88:3300/0 conn(0x7f12700b23b0 msgr2=0x7f12701426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:29.078 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.087+0000 7f12760b8640 1 --2- 172.21.15.49:0/3631365521 >> v2:172.21.15.88:3300/0 conn(0x7f12700b23b0 0x7f12701426a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:29.078 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.087+0000 7f12760b8640 1 -- 172.21.15.49:0/3631365521 >> v1:172.21.15.49:6789/0 conn(0x7f127013ffa0 legacy=0x7f12700728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:29.078 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.087+0000 7f12760b8640 10 monclient: _finish_hunting 0 2024-05-15T12:32:29.078 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.087+0000 7f12760b8640 1 monclient: found mon.noname-b 2024-05-15T12:32:29.078 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.087+0000 7f12760b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:29.078 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.087+0000 7f12760b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:29.078 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.087+0000 7f12760b8640 1 -- 172.21.15.49:0/3631365521 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12700a9ae0 con 0x7f1270142be0 2024-05-15T12:32:29.079 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.089+0000 7f12760b8640 1 --2- 172.21.15.49:0/3631365521 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127014cf30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1268009d90 tx=0x7f126802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae42d65d13064dd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:29.083 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 1 -- 172.21.15.49:0/3631365521 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268039070 con 0x7f1270142be0 2024-05-15T12:32:29.083 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:29.083 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:29.083 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 10 monclient: dump: 2024-05-15T12:32:29.083 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:29.083 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:29.083 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 10 monclient: _finish_auth 0 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 10 monclient: _check_auth_tickets 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 1 -- 172.21.15.49:0/3631365521 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126802cbc0 con 0x7f1270142be0 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f1277b1b640 5 monclient: authenticate success, global_id 4202 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f1277b1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f1277b1b640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:29.084 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 1 -- 172.21.15.49:0/3631365521 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268034440 con 0x7f1270142be0 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f1277b1b640 4 set_mon_vals no callback set 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.093+0000 7f12748b5640 10 monclient: dump: 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:29.085 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:29.088 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 10 monclient: shutdown 2024-05-15T12:32:29.088 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:29.088 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 1 -- 172.21.15.49:0/3631365521 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 msgr2=0x7f127014cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 1 --2- 172.21.15.49:0/3631365521 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127014cf30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1268009d90 tx=0x7f126802be10 comp rx=0 tx=0).stop 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 1 -- 172.21.15.49:0/3631365521 shutdown_connections 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 1 --2- 172.21.15.49:0/3631365521 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127014cf30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 1 --2- 172.21.15.49:0/3631365521 >> v2:172.21.15.88:3300/0 conn(0x7f12700b23b0 0x7f12701426a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 1 -- 172.21.15.49:0/3631365521 >> 172.21.15.49:0/3631365521 conn(0x7f12700b8390 msgr2=0x7f12700b87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 1 -- 172.21.15.49:0/3631365521 shutdown_connections 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 1 -- 172.21.15.49:0/3631365521 wait complete. 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 10 monclient: build_initial_monmap 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.094+0000 7f1277b1b640 10 monclient: monmap: 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:created 0.000000 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:29.089 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 1 librados: starting msgr at 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 1 librados: starting objecter 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 1 Processor -- start 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 1 -- start start 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 1 librados: setting wanted keys 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 1 librados: calling monclient init 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 10 monclient: init 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:32:29.090 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700b23b0 0x7f127013a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 10 monclient(hunting): picked mon.noname-a con 0x7f12700b23b0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f127013ffa0 0x7f12701376e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 10 monclient(hunting): picked mon.noname-b con 0x7f127013ffa0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127013c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 10 monclient(hunting): picked mon.noname-c con 0x7f1270142be0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.095+0000 7f1277b1b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f1277b1b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127014d470 con 0x7f12700b23b0 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f1277b1b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f1277b1b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12700aa250 con 0x7f127013ffa0 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f1277b1b640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f1277b1b640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127014d5f0 con 0x7f1270142be0 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f1277b1b640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f1277b1b640 10 monclient(hunting): authenticate will time out at 1507.258461s 2024-05-15T12:32:29.091 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f12760b8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f12750b6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f127013ffa0 0x7f12701376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f12760b8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127013c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:56186/0 (socket says 172.21.15.49:56186) 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f12760b8640 1 -- 172.21.15.49:0/2558686302 learned_addr learned my addr 172.21.15.49:0/2558686302 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f12760b8640 10 monclient(hunting): get_auth_request con 0x7f1270142be0 auth_method 0 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f12760b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f12760b8640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f12760b8640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.096+0000 7f12758b7640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700b23b0 0x7f127013a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12750b6640 10 monclient(hunting): get_auth_request con 0x7f127013ffa0 auth_method 0 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12750b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12750b6640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12750b6640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12758b7640 10 monclient(hunting): get_auth_request con 0x7f12700b23b0 auth_method 0 2024-05-15T12:32:29.092 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12758b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12758b7640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12758b7640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12760b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12760b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12760b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12750b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12750b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12750b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12758b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12758b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:29.093 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.097+0000 7f12758b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:29.094 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.099+0000 7f12760b8640 10 monclient(hunting): handle_auth_done global_id 4205 payload 824 2024-05-15T12:32:29.094 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.100+0000 7f12760b8640 1 -- 172.21.15.49:0/2558686302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f127013ffa0 msgr2=0x7f12701376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:29.094 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.100+0000 7f12760b8640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f127013ffa0 0x7f12701376e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:29.094 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.100+0000 7f12760b8640 1 -- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700b23b0 msgr2=0x7f127013a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:29.094 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.100+0000 7f12760b8640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700b23b0 0x7f127013a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:29.094 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.100+0000 7f12760b8640 10 monclient: _finish_hunting 0 2024-05-15T12:32:29.094 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.100+0000 7f12760b8640 1 monclient: found mon.noname-c 2024-05-15T12:32:29.095 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.100+0000 7f12760b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:29.095 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.100+0000 7f12760b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:29.095 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.100+0000 7f12760b8640 1 -- 172.21.15.49:0/2558686302 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127013bf20 con 0x7f1270142be0 2024-05-15T12:32:29.095 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.102+0000 7f12760b8640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127013c890 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1268009eb0 tx=0x7f1268004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:29.095 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.102+0000 7f12667fc640 10 client.?.objecter ms_handle_connect 0x7f1270142be0 2024-05-15T12:32:29.095 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.102+0000 7f12667fc640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:29.096 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.105+0000 7f12667fc640 1 -- 172.21.15.49:0/2558686302 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268039050 con 0x7f1270142be0 2024-05-15T12:32:29.096 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:29.096 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:29.096 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 10 monclient: dump: 2024-05-15T12:32:29.096 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:29.096 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:29.096 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:29.096 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 10 monclient: _finish_auth 0 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 10 monclient: _check_auth_tickets 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 1 -- 172.21.15.49:0/2558686302 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1268045070 con 0x7f1270142be0 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 1 -- 172.21.15.49:0/2558686302 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268009450 con 0x7f1270142be0 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f1277b1b640 5 monclient: authenticate success, global_id 4205 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f1277b1b640 10 monclient: _renew_subs 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f1277b1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:29.097 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f1277b1b640 1 -- 172.21.15.49:0/2558686302 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127013c190 con 0x7f1270142be0 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f1267fff640 4 set_mon_vals no callback set 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f12667fc640 10 monclient: dump: 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:29.098 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f1277b1b640 10 client.4205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:29.099 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f1277b1b640 10 monclient: _renew_subs 2024-05-15T12:32:29.099 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f1277b1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:29.099 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f1277b1b640 1 -- 172.21.15.49:0/2558686302 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f127013b150 con 0x7f1270142be0 2024-05-15T12:32:29.099 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.106+0000 7f1277b1b640 1 librados: init done 2024-05-15T12:32:29.099 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.107+0000 7f1277b1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:29.099 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.107+0000 7f1277b1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:29.099 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.107+0000 7f1277b1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:29.099 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.107+0000 7f1277b1b640 1 -- 172.21.15.49:0/2558686302 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1270068270 con 0x7f1270142be0 2024-05-15T12:32:29.116 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.126+0000 7f12667fc640 1 -- 172.21.15.49:0/2558686302 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f126803e850 con 0x7f1270142be0 2024-05-15T12:32:29.117 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.127+0000 7f12667fc640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f125406ce00 0x7f125406f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:29.117 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.127+0000 7f12758b7640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f125406ce00 0x7f125406f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:29.117 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.127+0000 7f12758b7640 10 monclient: get_auth_request con 0x7f125406ce00 auth_method 0 2024-05-15T12:32:29.117 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.127+0000 7f12667fc640 10 client.4205.objecter ms_handle_connect 0x7f125406ce00 2024-05-15T12:32:29.117 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.127+0000 7f12758b7640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f125406ce00 0x7f125406f2c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f126c007fd0 tx=0x7f126c007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:29.187 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:32:29.194+0000 ac84640 -1 osd.3 0 waiting for initial osdmap 2024-05-15T12:32:29.380 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.390+0000 7f12667fc640 1 -- 172.21.15.49:0/2558686302 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2743+0+0 (secure 0 0 0) 0x7f126807a0b0 con 0x7f1270142be0 2024-05-15T12:32:29.380 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.390+0000 7f12667fc640 10 client.4205.objecter ms_dispatch 0x7f1270000b90 osd_map(14..14 src has 1..14) v4 2024-05-15T12:32:29.381 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.390+0000 7f12667fc640 3 client.4205.objecter handle_osd_map got epochs [14,14] > 0 2024-05-15T12:32:29.381 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.390+0000 7f12667fc640 3 client.4205.objecter handle_osd_map decoding full epoch 14 2024-05-15T12:32:29.381 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.390+0000 7f12667fc640 20 client.4205.objecter dump_active .. 0 homeless 2024-05-15T12:32:29.381 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.390+0000 7f12667fc640 1 -- 172.21.15.49:0/2558686302 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f126807c010 con 0x7f1270142be0 2024-05-15T12:32:29.381 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.390+0000 7f12667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:29.381 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.390+0000 7f12667fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:29.585 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.595+0000 7f1277b1b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:29.585 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.595+0000 7f1277b1b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:29.585 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.595+0000 7f1277b1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:29.586 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.595+0000 7f1277b1b640 1 -- 172.21.15.49:0/2558686302 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12700aa3f0 con 0x7f1270142be0 2024-05-15T12:32:29.605 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:32:29.601+0000 10260640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:32:29.742 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.752+0000 7f12667fc640 1 -- 172.21.15.49:0/2558686302 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8029 (secure 0 0 0) 0x7f12700aa3f0 con 0x7f1270142be0 2024-05-15T12:32:29.742 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.752+0000 7f12667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:29.742 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.752+0000 7f12667fc640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:29.742 INFO:teuthology.misc.health.smithi049.stdout: 2024-05-15T12:32:29.742 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":14,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:32:28.752236+0000","last_up_change":"2024-05-15T12:32:28.752236+0000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6817","nonce":3384996234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6819","nonce":3384996234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6823","nonce":3384996234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6821","nonce":3384996234}]},"public_addr":"172.21.15.88:6817/3384996234","cluster_addr":"172.21.15.88:6819/3384996234","heartbeat_back_addr":"172.21.15.88:6823/3384996234","heartbeat_front_addr":"172.21.15.88:6821/3384996234","state":["exists","up"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6801","nonce":3138490191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6803","nonce":3138490191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6807","nonce":3138490191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6805","nonce":3138490191}]},"public_addr":"172.21.15.88:6801/3138490191","cluster_addr":"172.21.15.88:6803/3138490191","heartbeat_back_addr":"172.21.15.88:6807/3138490191","heartbeat_front_addr":"172.21.15.88:6805/3138490191","state":["exists","up"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:32:29.745 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 10 librados: watch_flush enter 2024-05-15T12:32:29.745 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 10 librados: watch_flush exit 2024-05-15T12:32:29.745 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 20 client.4205.objecter shutdown clearing up homeless session... 2024-05-15T12:32:29.745 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 10 client.4205.objecter successfully canceled tick 2024-05-15T12:32:29.745 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 1 -- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f125406ce00 msgr2=0x7f125406f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:29.745 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f125406ce00 0x7f125406f2c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f126c007fd0 tx=0x7f126c007e80 comp rx=0 tx=0).stop 2024-05-15T12:32:29.745 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 10 monclient: shutdown 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 1 -- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 msgr2=0x7f127013c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127013c890 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1268009eb0 tx=0x7f1268004a40 comp rx=0 tx=0).stop 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 1 -- 172.21.15.49:0/2558686302 shutdown_connections 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.754+0000 7f1277b1b640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1270142be0 0x7f127013c890 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.755+0000 7f1277b1b640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f125406ce00 0x7f125406f2c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.755+0000 7f1277b1b640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f127013ffa0 0x7f12701376e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.755+0000 7f1277b1b640 1 --2- 172.21.15.49:0/2558686302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700b23b0 0x7f127013a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.755+0000 7f1277b1b640 1 -- 172.21.15.49:0/2558686302 >> 172.21.15.49:0/2558686302 conn(0x7f12700b8390 msgr2=0x7f1270134cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.755+0000 7f1277b1b640 1 -- 172.21.15.49:0/2558686302 shutdown_connections 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.755+0000 7f1277b1b640 1 -- 172.21.15.49:0/2558686302 wait complete. 2024-05-15T12:32:29.746 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:29.755+0000 7f1277b1b640 1 librados: shutdown 2024-05-15T12:32:29.757 DEBUG:teuthology.misc:2 of 8 OSDs are up 2024-05-15T12:32:30.151 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:32:30.157+0000 c49a640 -1 osd.0 0 waiting for initial osdmap 2024-05-15T12:32:30.195 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:32:30.202+0000 ac84640 -1 osd.2 0 waiting for initial osdmap 2024-05-15T12:32:30.628 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:32:30.620+0000 10260640 -1 osd.0 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:32:30.695 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:32:30.691+0000 10260640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:32:34.929 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:34.875+0000 5dce500 -1 bdev(0x164e2e60 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 2 2024-05-15T12:32:35.029 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:34.941+0000 5dce500 -1 bdev(0x164e2e60 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 2 2024-05-15T12:32:35.029 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:34.956+0000 5dce500 -1 bdev(0x164e2e60 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-05-15T12:32:35.049 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:35.045+0000 5dce500 -1 bdev(0x164e2e60 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 2 2024-05-15T12:32:35.056 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:34.851+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-05-15T12:32:35.056 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:34.936+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-05-15T12:32:35.056 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:35.015+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-05-15T12:32:35.093 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:35.069+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-05-15T12:32:35.116 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:35.106+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-05-15T12:32:35.185 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:35.186+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-05-15T12:32:35.209 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:35.197+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-05-15T12:32:35.315 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:35.220+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-05-15T12:32:35.327 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:35.327+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-05-15T12:32:35.543 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:35.061+0000 5dce500 -1 bdev(0x164e2e60 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-05-15T12:32:35.550 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:35.375+0000 5dce500 -1 bdev(0x164e2e60 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-05-15T12:32:35.623 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:35.418+0000 5dce500 -1 bdev(0x164e2e60 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-05-15T12:32:35.624 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:35.451+0000 5dce500 -1 bdev(0x164e2e60 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-05-15T12:32:35.624 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:35.544+0000 5dce500 -1 bdev(0x164e2e60 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-05-15T12:32:35.624 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:35.562+0000 5dce500 -1 bdev(0x164e2e60 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-05-15T12:32:35.819 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:35.338+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-05-15T12:32:35.819 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:35.368+0000 5dce500 -1 bdev(0x164d67f0 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-05-15T12:32:36.563 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:32:36.653 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:36.660+0000 c49a640 -1 osd.5 0 waiting for initial osdmap 2024-05-15T12:32:36.668 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.679+0000 7f7f8647e640 10 monclient: get_monmap_and_config 2024-05-15T12:32:36.668 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.679+0000 7f7f8647e640 10 monclient: build_initial_monmap 2024-05-15T12:32:36.668 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.679+0000 7f7f8647e640 10 monclient: monmap: 2024-05-15T12:32:36.668 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:36.668 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:36.668 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:32:36.679921+0000 2024-05-15T12:32:36.668 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:32:36.679921+0000 2024-05-15T12:32:36.668 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:36.668 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:36.669 INFO:teuthology.misc.health.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:36.669 INFO:teuthology.misc.health.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:36.669 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:36.669 INFO:teuthology.misc.health.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:36.669 INFO:teuthology.misc.health.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:36.669 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:36.670 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 1 Processor -- start 2024-05-15T12:32:36.670 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 1 -- start start 2024-05-15T12:32:36.670 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 10 monclient: init 2024-05-15T12:32:36.670 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:36.670 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-05-15T12:32:36.670 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7f8013ffa0 0x7f7f800728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 10 monclient(hunting): picked mon.noname-c con 0x7f7f8013ffa0 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f800b23b0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7f8014af20 0x7f7f8014d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 10 monclient(hunting): picked mon.noname-a con 0x7f7f8014af20 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f800ab160 con 0x7f7f800b23b0 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f800ab440 con 0x7f7f8014af20 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f800a9ae0 con 0x7f7f8013ffa0 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:36.671 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.680+0000 7f7f8647e640 10 monclient(hunting): authenticate will time out at 1514.842391s 2024-05-15T12:32:36.676 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.687+0000 7f7f7ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7f8013ffa0 0x7f7f800728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:36.677 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.688+0000 7f7f7ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7f8013ffa0 0x7f7f800728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:34630/0 (socket says 172.21.15.49:34630) 2024-05-15T12:32:36.677 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.688+0000 7f7f7ffff640 1 -- 172.21.15.49:0/2672578238 learned_addr learned my addr 172.21.15.49:0/2672578238 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:36.677 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.688+0000 7f7f7ffff640 10 monclient(hunting): get_auth_request con 0x7f7f8013ffa0 auth_method 0 2024-05-15T12:32:36.677 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.688+0000 7f7f7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:36.677 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.688+0000 7f7f7ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:36.677 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.688+0000 7f7f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:36.678 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.688+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:36.678 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.688+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:36.678 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.688+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:36.680 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.691+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_done global_id 4252 payload 274 2024-05-15T12:32:36.680 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.691+0000 7f7f7ffff640 1 -- 172.21.15.49:0/2672578238 >> v2:172.21.15.49:3300/0 conn(0x7f7f8014af20 msgr2=0x7f7f8014d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:36.680 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.691+0000 7f7f7ffff640 1 --2- 172.21.15.49:0/2672578238 >> v2:172.21.15.49:3300/0 conn(0x7f7f8014af20 0x7f7f8014d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:36.680 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.691+0000 7f7f7ffff640 1 -- 172.21.15.49:0/2672578238 >> v1:172.21.15.49:6789/0 conn(0x7f7f800b23b0 legacy=0x7f7f801426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:36.680 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.691+0000 7f7f7ffff640 10 monclient: _finish_hunting 0 2024-05-15T12:32:36.680 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.691+0000 7f7f7ffff640 1 monclient: found mon.noname-c 2024-05-15T12:32:36.681 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.691+0000 7f7f7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:36.681 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.691+0000 7f7f7ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:36.681 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.691+0000 7f7f7ffff640 1 -- 172.21.15.49:0/2672578238 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f80072df0 con 0x7f7f8013ffa0 2024-05-15T12:32:36.683 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.694+0000 7f7f7ffff640 1 --2- 172.21.15.49:0/2672578238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f8013ffa0 0x7f7f800728b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f800b5b10 tx=0x7f7f7002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7fda6b0fba292ad server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:36.689 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 1 -- 172.21.15.49:0/2672578238 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7002e070 con 0x7f7f8013ffa0 2024-05-15T12:32:36.689 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:36.689 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:36.689 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 10 monclient: dump: 2024-05-15T12:32:36.690 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:36.690 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:36.690 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:36.690 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:36.690 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:36.690 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:36.690 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:36.690 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 10 monclient: _finish_auth 0 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 1 -- 172.21.15.49:0/2672578238 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f70037070 con 0x7f7f8013ffa0 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 1 -- 172.21.15.49:0/2672578238 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70031a00 con 0x7f7f8013ffa0 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:36.691 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f7effd640 10 monclient: dump: 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:36.692 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f8647e640 5 monclient: authenticate success, global_id 4252 2024-05-15T12:32:36.693 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f8647e640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:36.693 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.700+0000 7f7f8647e640 4 set_mon_vals no callback set 2024-05-15T12:32:36.696 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.701+0000 7f7f8647e640 10 monclient: shutdown 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.701+0000 7f7f8647e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.701+0000 7f7f8647e640 1 -- 172.21.15.49:0/2672578238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f8013ffa0 msgr2=0x7f7f800728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.701+0000 7f7f8647e640 1 --2- 172.21.15.49:0/2672578238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f8013ffa0 0x7f7f800728b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f800b5b10 tx=0x7f7f7002b880 comp rx=0 tx=0).stop 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.701+0000 7f7f8647e640 1 -- 172.21.15.49:0/2672578238 shutdown_connections 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.701+0000 7f7f8647e640 1 --2- 172.21.15.49:0/2672578238 >> v2:172.21.15.49:3300/0 conn(0x7f7f8014af20 0x7f7f8014d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.701+0000 7f7f8647e640 1 --2- 172.21.15.49:0/2672578238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f8013ffa0 0x7f7f800728b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.701+0000 7f7f8647e640 1 -- 172.21.15.49:0/2672578238 >> 172.21.15.49:0/2672578238 conn(0x7f7f800b8390 msgr2=0x7f7f800b87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.701+0000 7f7f8647e640 1 -- 172.21.15.49:0/2672578238 shutdown_connections 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.701+0000 7f7f8647e640 1 -- 172.21.15.49:0/2672578238 wait complete. 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.702+0000 7f7f8647e640 10 monclient: build_initial_monmap 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.702+0000 7f7f8647e640 10 monclient: monmap: 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:36.697 INFO:teuthology.misc.health.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:created 0.000000 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.702+0000 7f7f8647e640 1 librados: starting msgr at 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.702+0000 7f7f8647e640 1 librados: starting objecter 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.702+0000 7f7f8647e640 1 Processor -- start 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.703+0000 7f7f8647e640 1 -- start start 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.703+0000 7f7f8647e640 1 librados: setting wanted keys 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.703+0000 7f7f8647e640 1 librados: calling monclient init 2024-05-15T12:32:36.698 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.703+0000 7f7f8647e640 10 monclient: init 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.703+0000 7f7f8647e640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f800b23b0 0x7f7f80137180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 10 monclient(hunting): picked mon.noname-c con 0x7f7f800b23b0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f8013ffa0 0x7f7f801376c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 10 monclient(hunting): picked mon.noname-b con 0x7f7f8013ffa0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f8014af20 0x7f7f801db800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 10 monclient(hunting): picked mon.noname-a con 0x7f7f8014af20 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f80142db0 con 0x7f7f8014af20 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f800aa210 con 0x7f7f8013ffa0 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f80142f30 con 0x7f7f800b23b0 2024-05-15T12:32:36.699 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f8647e640 10 monclient(hunting): authenticate will time out at 1514.865539s 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.704+0000 7f7f7ffff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f800b23b0 0x7f7f80137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7ffff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f800b23b0 0x7f7f80137180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:48456/0 (socket says 172.21.15.49:48456) 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7ffff640 1 -- 172.21.15.49:0/2896114127 learned_addr learned my addr 172.21.15.49:0/2896114127 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7ffff640 10 monclient(hunting): get_auth_request con 0x7f7f800b23b0 auth_method 0 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7f7fe640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f8013ffa0 0x7f7f801376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7f7fe640 10 monclient(hunting): get_auth_request con 0x7f7f8013ffa0 auth_method 0 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:36.700 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:36.701 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.705+0000 7f7f7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:36.701 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.706+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:36.701 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.706+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:36.701 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.706+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:36.701 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.708+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_done global_id 4208 payload 824 2024-05-15T12:32:36.701 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.708+0000 7f7f7ffff640 1 -- 172.21.15.49:0/2896114127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f8013ffa0 msgr2=0x7f7f801376c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.708+0000 7f7f7ffff640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f8013ffa0 0x7f7f801376c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.708+0000 7f7f7ffff640 1 -- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f8014af20 msgr2=0x7f7f801db800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.708+0000 7f7f7ffff640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f8014af20 0x7f7f801db800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.708+0000 7f7f7ffff640 10 monclient: _finish_hunting 0 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.708+0000 7f7f7ffff640 1 monclient: found mon.noname-c 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.708+0000 7f7f7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.709+0000 7f7f7ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.709+0000 7f7f7ffff640 1 -- 172.21.15.49:0/2896114127 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f80137c00 con 0x7f7f800b23b0 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.712+0000 7f7f7ffff640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f800b23b0 0x7f7f80137180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7f70000c00 tx=0x7f7f700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.712+0000 7f7f7cff9640 10 client.?.objecter ms_handle_connect 0x7f7f800b23b0 2024-05-15T12:32:36.702 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.712+0000 7f7f7cff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:36.705 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 1 -- 172.21.15.49:0/2896114127 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70032880 con 0x7f7f800b23b0 2024-05-15T12:32:36.705 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:36.705 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:36.705 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 10 monclient: dump: 2024-05-15T12:32:36.705 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:36.705 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:36.705 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 10 monclient: _finish_auth 0 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 10 monclient: _check_auth_tickets 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:36.706 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 1 -- 172.21.15.49:0/2896114127 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f70032a20 con 0x7f7f800b23b0 2024-05-15T12:32:36.707 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:36.707 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 1 -- 172.21.15.49:0/2896114127 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70032e10 con 0x7f7f800b23b0 2024-05-15T12:32:36.707 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f8647e640 5 monclient: authenticate success, global_id 4208 2024-05-15T12:32:36.707 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:36.707 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:36.707 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7cff9640 10 monclient: dump: 2024-05-15T12:32:36.707 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:36.707 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:36.707 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:36.707 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f8647e640 10 monclient: _renew_subs 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f8647e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f8647e640 1 -- 172.21.15.49:0/2896114127 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f800b1c00 con 0x7f7f800b23b0 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f7e7fc640 4 set_mon_vals no callback set 2024-05-15T12:32:36.708 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f8647e640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:36.709 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f8647e640 10 monclient: _renew_subs 2024-05-15T12:32:36.709 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f8647e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:36.709 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f8647e640 1 -- 172.21.15.49:0/2896114127 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f800aff00 con 0x7f7f800b23b0 2024-05-15T12:32:36.709 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.716+0000 7f7f8647e640 1 librados: init done 2024-05-15T12:32:36.710 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:36.713+0000 ac84640 -1 osd.7 0 waiting for initial osdmap 2024-05-15T12:32:36.710 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.717+0000 7f7f8647e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:36.710 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.717+0000 7f7f8647e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:36.710 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.717+0000 7f7f8647e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:36.710 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.717+0000 7f7f8647e640 1 -- 172.21.15.49:0/2896114127 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f800b0130 con 0x7f7f800b23b0 2024-05-15T12:32:36.733 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.744+0000 7f7f7cff9640 1 -- 172.21.15.49:0/2896114127 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f7f700453f0 con 0x7f7f800b23b0 2024-05-15T12:32:36.733 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.744+0000 7f7f7cff9640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7f6c06cdd0 0x7f7f6c06f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:36.733 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.744+0000 7f7f7f7fe640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7f6c06cdd0 0x7f7f6c06f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:36.733 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.744+0000 7f7f7f7fe640 10 monclient: get_auth_request con 0x7f7f6c06cdd0 auth_method 0 2024-05-15T12:32:36.734 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.745+0000 7f7f7cff9640 10 client.4208.objecter ms_handle_connect 0x7f7f6c06cdd0 2024-05-15T12:32:36.734 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.745+0000 7f7f7f7fe640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7f6c06cdd0 0x7f7f6c06f290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f800b1420 tx=0x7f7f74009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:36.737 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.748+0000 7f7f7cff9640 1 -- 172.21.15.49:0/2896114127 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4017+0+0 (secure 0 0 0) 0x7f7f70078f80 con 0x7f7f800b23b0 2024-05-15T12:32:36.738 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.748+0000 7f7f7cff9640 10 client.4208.objecter ms_dispatch 0x7f7f80000b90 osd_map(18..18 src has 1..18) v4 2024-05-15T12:32:36.738 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.748+0000 7f7f7cff9640 3 client.4208.objecter handle_osd_map got epochs [18,18] > 0 2024-05-15T12:32:36.738 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.748+0000 7f7f7cff9640 3 client.4208.objecter handle_osd_map decoding full epoch 18 2024-05-15T12:32:36.738 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:36.749+0000 7f7f7cff9640 20 client.4208.objecter dump_active .. 0 homeless 2024-05-15T12:32:37.000 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.011+0000 7f7f7cff9640 1 -- 172.21.15.49:0/2896114127 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7f800b1c00 con 0x7f7f800b23b0 2024-05-15T12:32:37.000 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.011+0000 7f7f7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:37.000 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.011+0000 7f7f7cff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:37.086 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:32:37.083+0000 14268640 -1 osd.5 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:32:37.129 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T12:32:37.119+0000 14268640 -1 osd.7 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:32:37.223 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.234+0000 7f7f8647e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:37.223 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.234+0000 7f7f8647e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:37.223 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.234+0000 7f7f8647e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:37.223 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.234+0000 7f7f8647e640 1 -- 172.21.15.49:0/2896114127 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f800b1eb0 con 0x7f7f800b23b0 2024-05-15T12:32:37.419 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.430+0000 7f7f7cff9640 1 -- 172.21.15.49:0/2896114127 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+11113 (secure 0 0 0) 0x7f7f70040070 con 0x7f7f800b23b0 2024-05-15T12:32:37.420 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.430+0000 7f7f7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:37.420 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.430+0000 7f7f7cff9640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:37.420 INFO:teuthology.misc.health.smithi049.stdout: 2024-05-15T12:32:37.420 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":18,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:32:33.120098+0000","last_up_change":"2024-05-15T12:32:32.029518+0000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:32:30.595515+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6819","nonce":3376151072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6821","nonce":3376151072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6825","nonce":3376151072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6823","nonce":3376151072}]},"public_addr":"172.21.15.49:6819/3376151072","cluster_addr":"172.21.15.49:6821/3376151072","heartbeat_back_addr":"172.21.15.49:6825/3376151072","heartbeat_front_addr":"172.21.15.49:6823/3376151072","state":["exists","up"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6827","nonce":2361777220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6829","nonce":2361777220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6833","nonce":2361777220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6831","nonce":2361777220}]},"public_addr":"172.21.15.49:6827/2361777220","cluster_addr":"172.21.15.49:6829/2361777220","heartbeat_back_addr":"172.21.15.49:6833/2361777220","heartbeat_front_addr":"172.21.15.49:6831/2361777220","state":["exists","up"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6803","nonce":945312262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6805","nonce":945312262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6809","nonce":945312262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6807","nonce":945312262}]},"public_addr":"172.21.15.49:6803/945312262","cluster_addr":"172.21.15.49:6805/945312262","heartbeat_back_addr":"172.21.15.49:6809/945312262","heartbeat_front_addr":"172.21.15.49:6807/945312262","state":["exists","up"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6817","nonce":3384996234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6819","nonce":3384996234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6823","nonce":3384996234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6821","nonce":3384996234}]},"public_addr":"172.21.15.88:6817/3384996234","cluster_addr":"172.21.15.88:6819/3384996234","heartbeat_back_addr":"172.21.15.88:6823/3384996234","heartbeat_front_addr":"172.21.15.88:6821/3384996234","state":["exists","up"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6801","nonce":3138490191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6803","nonce":3138490191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6807","nonce":3138490191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6805","nonce":3138490191}]},"public_addr":"172.21.15.88:6801/3138490191","cluster_addr":"172.21.15.88:6803/3138490191","heartbeat_back_addr":"172.21.15.88:6807/3138490191","heartbeat_front_addr":"172.21.15.88:6805/3138490191","state":["exists","up"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:28.806234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:28.402190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:27.582000+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:25.805322+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.001531+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:32:37.423 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 10 librados: watch_flush enter 2024-05-15T12:32:37.423 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 10 librados: watch_flush exit 2024-05-15T12:32:37.423 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 20 client.4208.objecter shutdown clearing up homeless session... 2024-05-15T12:32:37.423 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 10 client.4208.objecter successfully canceled tick 2024-05-15T12:32:37.424 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 -- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7f6c06cdd0 msgr2=0x7f7f6c06f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:37.424 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7f6c06cdd0 0x7f7f6c06f290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f800b1420 tx=0x7f7f74009040 comp rx=0 tx=0).stop 2024-05-15T12:32:37.424 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 10 monclient: shutdown 2024-05-15T12:32:37.424 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:37.424 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 -- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f800b23b0 msgr2=0x7f7f80137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:37.424 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f800b23b0 0x7f7f80137180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7f70000c00 tx=0x7f7f700046c0 comp rx=0 tx=0).stop 2024-05-15T12:32:37.424 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 -- 172.21.15.49:0/2896114127 shutdown_connections 2024-05-15T12:32:37.425 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f8014af20 0x7f7f801db800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:37.425 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7f6c06cdd0 0x7f7f6c06f290 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:37.425 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f8013ffa0 0x7f7f801376c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:37.425 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 --2- 172.21.15.49:0/2896114127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f800b23b0 0x7f7f80137180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:37.425 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 -- 172.21.15.49:0/2896114127 >> 172.21.15.49:0/2896114127 conn(0x7f7f800b8390 msgr2=0x7f7f800b7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:37.425 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 -- 172.21.15.49:0/2896114127 shutdown_connections 2024-05-15T12:32:37.425 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 -- 172.21.15.49:0/2896114127 wait complete. 2024-05-15T12:32:37.426 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:37.432+0000 7f7f8647e640 1 librados: shutdown 2024-05-15T12:32:37.433 DEBUG:teuthology.misc:5 of 8 OSDs are up 2024-05-15T12:32:38.144 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.078+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.145 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.144+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.187 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.176+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-05-15T12:32:38.187 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.187+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.335 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.200+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.349 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.321+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.453 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.346+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.454 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.360+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.466 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.394+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.561 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.449+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.570 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.465+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.570 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.478+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.571 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.573+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.614 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.585+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.725 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.626+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.726 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.640+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:38.799 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.783+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:39.074 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:38.843+0000 5dce500 -1 bdev(0x164b0e50 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-15T12:32:40.024 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:40.031+0000 c49a640 -1 osd.1 0 waiting for initial osdmap 2024-05-15T12:32:40.300 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:01:09.509 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:32:40.387 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:01:09.597 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:32:40.394 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:01:09.560 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:32:40.408 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:01:09.617 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:32:40.450 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:01:09.617 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:32:40.475 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:01:09.684 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:32:40.564 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:32:40.560+0000 14268640 -1 osd.1 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:32:40.667 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:01:09.877 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:32:40.722 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:01:09.890 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:32:40.764 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:01:09.973 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:32:44.241 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-15T12:32:44.335 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.346+0000 7f844411c640 10 monclient: get_monmap_and_config 2024-05-15T12:32:44.335 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.346+0000 7f844411c640 10 monclient: build_initial_monmap 2024-05-15T12:32:44.335 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.346+0000 7f844411c640 10 monclient: monmap: 2024-05-15T12:32:44.335 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:44.335 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:44.335 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:32:44.346820+0000 2024-05-15T12:32:44.336 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:32:44.346820+0000 2024-05-15T12:32:44.336 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:44.336 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:44.336 INFO:teuthology.misc.health.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:44.336 INFO:teuthology.misc.health.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:44.336 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:44.336 INFO:teuthology.misc.health.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:44.336 INFO:teuthology.misc.health.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:44.336 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:44.338 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.346+0000 7f844411c640 1 Processor -- start 2024-05-15T12:32:44.338 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 1 -- start start 2024-05-15T12:32:44.338 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 10 monclient: init 2024-05-15T12:32:44.338 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:44.338 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-05-15T12:32:44.338 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b51e0 0x7f843c13f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:44.338 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 10 monclient(hunting): picked mon.noname-b con 0x7f843c0b51e0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:44.338 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f843c0b23d0 0x7f843c13fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:44.338 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 10 monclient(hunting): picked mon.noname-a con 0x7f843c0b23d0 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f843c142770 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843c0aa230 con 0x7f843c142770 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f843c0b55c0 con 0x7f843c0b23d0 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843c140330 con 0x7f843c0b51e0 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.347+0000 7f844411c640 10 monclient(hunting): authenticate will time out at 1522.509248s 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f8441eb8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b51e0 0x7f843c13f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f84416b7640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f843c0b23d0 0x7f843c13fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f8441eb8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b51e0 0x7f843c13f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:48470/0 (socket says 172.21.15.49:48470) 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f84416b7640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f843c0b23d0 0x7f843c13fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49064/0 (socket says 172.21.15.49:49064) 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f8441eb8640 1 -- 172.21.15.49:0/243033696 learned_addr learned my addr 172.21.15.49:0/243033696 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:44.339 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f8441eb8640 10 monclient(hunting): get_auth_request con 0x7f843c0b51e0 auth_method 0 2024-05-15T12:32:44.340 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f8441eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:44.340 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f8441eb8640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:44.340 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f8441eb8640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:44.340 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f84416b7640 10 monclient(hunting): get_auth_request con 0x7f843c0b23d0 auth_method 0 2024-05-15T12:32:44.340 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f84416b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:44.340 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f84416b7640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:44.340 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.348+0000 7f84416b7640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:44.340 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.349+0000 7f8441eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:44.340 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.349+0000 7f8441eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:44.340 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.349+0000 7f8441eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:44.341 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.350+0000 7f84416b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:44.341 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.350+0000 7f84416b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:44.341 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.350+0000 7f84416b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:44.341 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.351+0000 7f8441eb8640 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2024-05-15T12:32:44.341 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.351+0000 7f8441eb8640 1 -- 172.21.15.49:0/243033696 >> v2:172.21.15.49:3300/0 conn(0x7f843c0b23d0 msgr2=0x7f843c13fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:44.341 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.351+0000 7f8441eb8640 1 --2- 172.21.15.49:0/243033696 >> v2:172.21.15.49:3300/0 conn(0x7f843c0b23d0 0x7f843c13fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:44.342 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.351+0000 7f8441eb8640 1 -- 172.21.15.49:0/243033696 >> v1:172.21.15.49:6789/0 conn(0x7f843c142770 legacy=0x7f843c142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:44.342 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.351+0000 7f8441eb8640 10 monclient: _finish_hunting 0 2024-05-15T12:32:44.342 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.351+0000 7f8441eb8640 1 monclient: found mon.noname-b 2024-05-15T12:32:44.342 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.351+0000 7f8441eb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:44.342 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.351+0000 7f8441eb8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:44.342 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.351+0000 7f8441eb8640 1 -- 172.21.15.49:0/243033696 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c143310 con 0x7f843c0b51e0 2024-05-15T12:32:44.342 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.353+0000 7f8441eb8640 1 --2- 172.21.15.49:0/243033696 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b51e0 0x7f843c13f8b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f84340098a0 tx=0x7f843402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98db8882065263ea server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.356+0000 7f8440eb6640 1 -- 172.21.15.49:0/243033696 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843402f830 con 0x7f843c0b51e0 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.356+0000 7f8440eb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.356+0000 7f8440eb6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.356+0000 7f8440eb6640 10 monclient: dump: 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:44.346 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:44.347 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:44.347 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.356+0000 7f8440eb6640 10 monclient: _finish_auth 0 2024-05-15T12:32:44.347 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.356+0000 7f8440eb6640 10 monclient: _check_auth_tickets 2024-05-15T12:32:44.347 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.356+0000 7f8440eb6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:44.347 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.356+0000 7f844411c640 5 monclient: authenticate success, global_id 4211 2024-05-15T12:32:44.347 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.356+0000 7f844411c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:44.349 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.357+0000 7f8440eb6640 1 -- 172.21.15.49:0/243033696 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843402f9d0 con 0x7f843c0b51e0 2024-05-15T12:32:44.349 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.357+0000 7f8440eb6640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:44.349 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.357+0000 7f8440eb6640 1 -- 172.21.15.49:0/243033696 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843402fda0 con 0x7f843c0b51e0 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.357+0000 7f844411c640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.357+0000 7f844411c640 4 set_mon_vals no callback set 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.357+0000 7f8440eb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.357+0000 7f8440eb6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.357+0000 7f8440eb6640 10 monclient: dump: 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:44.350 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.358+0000 7f844411c640 10 monclient: shutdown 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.358+0000 7f844411c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.358+0000 7f844411c640 1 -- 172.21.15.49:0/243033696 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b51e0 msgr2=0x7f843c13f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.358+0000 7f844411c640 1 --2- 172.21.15.49:0/243033696 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b51e0 0x7f843c13f8b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f84340098a0 tx=0x7f843402ba10 comp rx=0 tx=0).stop 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.358+0000 7f844411c640 1 -- 172.21.15.49:0/243033696 shutdown_connections 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.358+0000 7f844411c640 1 --2- 172.21.15.49:0/243033696 >> v2:172.21.15.49:3300/0 conn(0x7f843c0b23d0 0x7f843c13fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.358+0000 7f844411c640 1 --2- 172.21.15.49:0/243033696 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b51e0 0x7f843c13f8b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.358+0000 7f844411c640 1 -- 172.21.15.49:0/243033696 >> 172.21.15.49:0/243033696 conn(0x7f843c067020 msgr2=0x7f843c067430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.358+0000 7f844411c640 1 -- 172.21.15.49:0/243033696 shutdown_connections 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.358+0000 7f844411c640 1 -- 172.21.15.49:0/243033696 wait complete. 2024-05-15T12:32:44.351 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.359+0000 7f844411c640 10 monclient: build_initial_monmap 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.359+0000 7f844411c640 10 monclient: monmap: 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:epoch 0 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:created 0.000000 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.359+0000 7f844411c640 1 librados: starting msgr at 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.359+0000 7f844411c640 1 librados: starting objecter 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.359+0000 7f844411c640 1 Processor -- start 2024-05-15T12:32:44.352 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.359+0000 7f844411c640 1 -- start start 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.359+0000 7f844411c640 1 librados: setting wanted keys 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.359+0000 7f844411c640 1 librados: calling monclient init 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.359+0000 7f844411c640 10 monclient: init 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b23d0 0x7f843c0b8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 10 monclient(hunting): picked mon.noname-c con 0x7f843c0b23d0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f843c0b51e0 0x7f843c0b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 10 monclient(hunting): picked mon.noname-a con 0x7f843c0b51e0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f843c142770 0x7f843c0b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 10 monclient(hunting): picked mon.noname-b con 0x7f843c142770 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c0a9f60 con 0x7f843c0b51e0 2024-05-15T12:32:44.353 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:44.354 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c0a9de0 con 0x7f843c142770 2024-05-15T12:32:44.354 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:44.354 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843c14d000 con 0x7f843c0b23d0 2024-05-15T12:32:44.354 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:44.354 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f844411c640 10 monclient(hunting): authenticate will time out at 1522.521868s 2024-05-15T12:32:44.354 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f84416b7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f843c0b51e0 0x7f843c0b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:44.354 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.360+0000 7f8441eb8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b23d0 0x7f843c0b8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84416b7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f843c0b51e0 0x7f843c0b5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49080/0 (socket says 172.21.15.49:49080) 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84416b7640 1 -- 172.21.15.49:0/326745599 learned_addr learned my addr 172.21.15.49:0/326745599 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84416b7640 10 monclient(hunting): get_auth_request con 0x7f843c0b51e0 auth_method 0 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84416b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84416b7640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84416b7640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84426b9640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f843c142770 0x7f843c0b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f8441eb8640 10 monclient(hunting): get_auth_request con 0x7f843c0b23d0 auth_method 0 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f8441eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f8441eb8640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f8441eb8640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84426b9640 10 monclient(hunting): get_auth_request con 0x7f843c142770 auth_method 0 2024-05-15T12:32:44.356 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84426b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84426b9640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84426b9640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84416b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84416b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f84416b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f8441eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f8441eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.361+0000 7f8441eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.362+0000 7f84426b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.362+0000 7f84426b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.362+0000 7f84426b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.364+0000 7f84416b7640 10 monclient(hunting): handle_auth_done global_id 4248 payload 824 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.364+0000 7f84416b7640 1 -- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b23d0 msgr2=0x7f843c0b8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.364+0000 7f84416b7640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b23d0 0x7f843c0b8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:44.357 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.364+0000 7f84416b7640 1 -- 172.21.15.49:0/326745599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f843c142770 msgr2=0x7f843c0b6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:44.358 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.364+0000 7f84416b7640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f843c142770 0x7f843c0b6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:44.358 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.364+0000 7f84416b7640 10 monclient: _finish_hunting 0 2024-05-15T12:32:44.358 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.364+0000 7f84416b7640 1 monclient: found mon.noname-a 2024-05-15T12:32:44.358 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.364+0000 7f84416b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:44.358 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.364+0000 7f84416b7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:44.358 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.364+0000 7f84416b7640 1 -- 172.21.15.49:0/326745599 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c0b7cd0 con 0x7f843c0b51e0 2024-05-15T12:32:44.358 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.368+0000 7f84416b7640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f843c0b51e0 0x7f843c0b5f40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f843000efc0 tx=0x7f843000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:44.358 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.368+0000 7f843a7fc640 10 client.?.objecter ms_handle_connect 0x7f843c0b51e0 2024-05-15T12:32:44.358 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.368+0000 7f843a7fc640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:44.364 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 1 -- 172.21.15.49:0/326745599 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430018020 con 0x7f843c0b51e0 2024-05-15T12:32:44.364 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:44.364 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:44.364 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 10 monclient: dump: 2024-05-15T12:32:44.364 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:44.364 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:44.364 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:44.364 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:44.364 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 10 monclient: _finish_auth 0 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 1 -- 172.21.15.49:0/326745599 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8430014480 con 0x7f843c0b51e0 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 1 -- 172.21.15.49:0/326745599 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430014850 con 0x7f843c0b51e0 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f844411c640 5 monclient: authenticate success, global_id 4248 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f844411c640 10 monclient: _renew_subs 2024-05-15T12:32:44.365 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f844411c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f844411c640 1 -- 172.21.15.49:0/326745599 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843c0b7fb0 con 0x7f843c0b51e0 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843a7fc640 10 monclient: dump: 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:epoch 1 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:44.366 INFO:teuthology.misc.health.smithi049.stderr: 2024-05-15T12:32:44.367 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f843bfff640 4 set_mon_vals no callback set 2024-05-15T12:32:44.367 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f844411c640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:44.367 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f844411c640 10 monclient: _renew_subs 2024-05-15T12:32:44.367 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f844411c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:44.367 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f844411c640 1 -- 172.21.15.49:0/326745599 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843c0b6d90 con 0x7f843c0b51e0 2024-05-15T12:32:44.367 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.375+0000 7f844411c640 1 librados: init done 2024-05-15T12:32:44.367 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.376+0000 7f844411c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:44.367 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.376+0000 7f844411c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:44.367 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.376+0000 7f844411c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:44.368 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.376+0000 7f844411c640 1 -- 172.21.15.49:0/326745599 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f843c04f0e0 con 0x7f843c0b51e0 2024-05-15T12:32:44.639 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.650+0000 7f843a7fc640 1 -- 172.21.15.49:0/326745599 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f8430005d10 con 0x7f843c0b51e0 2024-05-15T12:32:44.639 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.650+0000 7f843a7fc640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f842806ce20 0x7f842806f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:44.639 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.650+0000 7f8441eb8640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f842806ce20 0x7f842806f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:44.639 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.650+0000 7f8441eb8640 10 monclient: get_auth_request con 0x7f842806ce20 auth_method 0 2024-05-15T12:32:44.640 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.651+0000 7f843a7fc640 10 client.4248.objecter ms_handle_connect 0x7f842806ce20 2024-05-15T12:32:44.640 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.651+0000 7f8441eb8640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f842806ce20 0x7f842806f2e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8434036000 tx=0x7f8434008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:44.653 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.664+0000 7f843a7fc640 1 -- 172.21.15.49:0/326745599 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f843005fe00 con 0x7f843c0b51e0 2024-05-15T12:32:44.653 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.664+0000 7f843a7fc640 10 client.4248.objecter ms_dispatch 0x7f843c000b90 osd_map(22..22 src has 1..22) v4 2024-05-15T12:32:44.653 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.664+0000 7f843a7fc640 3 client.4248.objecter handle_osd_map got epochs [22,22] > 0 2024-05-15T12:32:44.653 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.664+0000 7f843a7fc640 3 client.4248.objecter handle_osd_map decoding full epoch 22 2024-05-15T12:32:44.654 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.664+0000 7f843a7fc640 20 client.4248.objecter dump_active .. 0 homeless 2024-05-15T12:32:44.654 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.664+0000 7f843a7fc640 1 -- 172.21.15.49:0/326745599 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8430022070 con 0x7f843c0b51e0 2024-05-15T12:32:44.654 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.664+0000 7f843a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:44.654 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.664+0000 7f843a7fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:44.846 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.857+0000 7f844411c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:44.847 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.857+0000 7f844411c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:44.847 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.857+0000 7f844411c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:44.847 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.857+0000 7f844411c640 1 -- 172.21.15.49:0/326745599 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f843c0a94d0 con 0x7f843c0b51e0 2024-05-15T12:32:44.964 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.974+0000 7f843a7fc640 1 -- 172.21.15.49:0/326745599 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+12837 (secure 0 0 0) 0x7f8430010040 con 0x7f843c0b51e0 2024-05-15T12:32:44.964 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.975+0000 7f843a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:44.964 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.975+0000 7f843a7fc640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:44.964 INFO:teuthology.misc.health.smithi049.stdout: 2024-05-15T12:32:44.964 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":22,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:32:42.267746+0000","last_up_change":"2024-05-15T12:32:41.022082+0000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:32:30.595515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6819","nonce":3376151072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6821","nonce":3376151072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6825","nonce":3376151072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6823","nonce":3376151072}]},"public_addr":"172.21.15.49:6819/3376151072","cluster_addr":"172.21.15.49:6821/3376151072","heartbeat_back_addr":"172.21.15.49:6825/3376151072","heartbeat_front_addr":"172.21.15.49:6823/3376151072","state":["exists","up"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6811","nonce":3250791401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6813","nonce":3250791401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6817","nonce":3250791401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6815","nonce":3250791401}]},"public_addr":"172.21.15.49:6811/3250791401","cluster_addr":"172.21.15.49:6813/3250791401","heartbeat_back_addr":"172.21.15.49:6817/3250791401","heartbeat_front_addr":"172.21.15.49:6815/3250791401","state":["exists","up"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6827","nonce":2361777220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6829","nonce":2361777220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6833","nonce":2361777220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6831","nonce":2361777220}]},"public_addr":"172.21.15.49:6827/2361777220","cluster_addr":"172.21.15.49:6829/2361777220","heartbeat_back_addr":"172.21.15.49:6833/2361777220","heartbeat_front_addr":"172.21.15.49:6831/2361777220","state":["exists","up"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6803","nonce":945312262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6805","nonce":945312262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6809","nonce":945312262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6807","nonce":945312262}]},"public_addr":"172.21.15.49:6803/945312262","cluster_addr":"172.21.15.49:6805/945312262","heartbeat_back_addr":"172.21.15.49:6809/945312262","heartbeat_front_addr":"172.21.15.49:6807/945312262","state":["exists","up"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6817","nonce":3384996234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6819","nonce":3384996234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6823","nonce":3384996234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6821","nonce":3384996234}]},"public_addr":"172.21.15.88:6817/3384996234","cluster_addr":"172.21.15.88:6819/3384996234","heartbeat_back_addr":"172.21.15.88:6823/3384996234","heartbeat_front_addr":"172.21.15.88:6821/3384996234","state":["exists","up"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6809","nonce":3085185190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6811","nonce":3085185190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6815","nonce":3085185190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6813","nonce":3085185190}]},"public_addr":"172.21.15.88:6809/3085185190","cluster_addr":"172.21.15.88:6811/3085185190","heartbeat_back_addr":"172.21.15.88:6815/3085185190","heartbeat_front_addr":"172.21.15.88:6813/3085185190","state":["exists","up"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6801","nonce":3138490191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6803","nonce":3138490191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6807","nonce":3138490191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6805","nonce":3138490191}]},"public_addr":"172.21.15.88:6801/3138490191","cluster_addr":"172.21.15.88:6803/3138490191","heartbeat_back_addr":"172.21.15.88:6807/3138490191","heartbeat_front_addr":"172.21.15.88:6805/3138490191","state":["exists","up"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6825","nonce":2037618931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6827","nonce":2037618931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6831","nonce":2037618931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6829","nonce":2037618931}]},"public_addr":"172.21.15.88:6825/2037618931","cluster_addr":"172.21.15.88:6827/2037618931","heartbeat_back_addr":"172.21.15.88:6831/2037618931","heartbeat_front_addr":"172.21.15.88:6829/2037618931","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:28.806234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:29.001705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:28.402190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:27.582000+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:25.805322+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.269314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.001531+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.945037+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:32:44.967 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 10 librados: watch_flush enter 2024-05-15T12:32:44.967 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 10 librados: watch_flush exit 2024-05-15T12:32:44.967 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 20 client.4248.objecter shutdown clearing up homeless session... 2024-05-15T12:32:44.967 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 10 client.4248.objecter successfully canceled tick 2024-05-15T12:32:44.967 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 -- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f842806ce20 msgr2=0x7f842806f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:44.967 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f842806ce20 0x7f842806f2e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8434036000 tx=0x7f8434008030 comp rx=0 tx=0).stop 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 10 monclient: shutdown 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 -- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f843c0b51e0 msgr2=0x7f843c0b5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f843c0b51e0 0x7f843c0b5f40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f843000efc0 tx=0x7f843000c490 comp rx=0 tx=0).stop 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 -- 172.21.15.49:0/326745599 shutdown_connections 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f842806ce20 0x7f842806f2e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f843c142770 0x7f843c0b6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f843c0b51e0 0x7f843c0b5f40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 --2- 172.21.15.49:0/326745599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f843c0b23d0 0x7f843c0b8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 -- 172.21.15.49:0/326745599 >> 172.21.15.49:0/326745599 conn(0x7f843c067020 msgr2=0x7f843c135b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 -- 172.21.15.49:0/326745599 shutdown_connections 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 -- 172.21.15.49:0/326745599 wait complete. 2024-05-15T12:32:44.968 INFO:teuthology.misc.health.smithi049.stderr:2024-05-15T12:32:44.977+0000 7f844411c640 1 librados: shutdown 2024-05-15T12:32:44.976 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-15T12:32:44.976 INFO:tasks.ceph:Creating RBD pool 2024-05-15T12:32:44.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool create rbd 8 2024-05-15T12:32:45.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.093+0000 7eff5dbb3640 10 monclient: get_monmap_and_config 2024-05-15T12:32:45.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.093+0000 7eff5dbb3640 10 monclient: build_initial_monmap 2024-05-15T12:32:45.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.093+0000 7eff5dbb3640 10 monclient: monmap: 2024-05-15T12:32:45.082 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:45.082 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:45.083 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:45.094105+0000 2024-05-15T12:32:45.083 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:45.094105+0000 2024-05-15T12:32:45.083 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:45.083 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:45.083 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:45.083 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:45.083 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:45.083 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:45.083 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:45.083 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:45.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.094+0000 7eff5dbb3640 1 Processor -- start 2024-05-15T12:32:45.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.094+0000 7eff5dbb3640 1 -- start start 2024-05-15T12:32:45.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.094+0000 7eff5dbb3640 10 monclient: init 2024-05-15T12:32:45.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.094+0000 7eff5dbb3640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:45.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-05-15T12:32:45.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7eff580b5350 0x7eff580a9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 10 monclient(hunting): picked mon.noname-a con 0x7eff580b5350 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff5813f8f0 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff58145b60 0x7eff58145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 10 monclient(hunting): picked mon.noname-c con 0x7eff58145b60 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff58146530 con 0x7eff5813f8f0 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff580b5730 con 0x7eff580b5350 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff580ab120 con 0x7eff58145b60 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff5dbb3640 10 monclient(hunting): authenticate will time out at 1523.256516s 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff577fe640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7eff580b5350 0x7eff580a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff57fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff58145b60 0x7eff58145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:45.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff57fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff58145b60 0x7eff58145ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:34888/0 (socket says 172.21.15.49:34888) 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff57fff640 1 -- 172.21.15.49:0/3643586597 learned_addr learned my addr 172.21.15.49:0/3643586597 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff57fff640 10 monclient(hunting): get_auth_request con 0x7eff58145b60 auth_method 0 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff57fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.095+0000 7eff57fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.096+0000 7eff577fe640 10 monclient(hunting): get_auth_request con 0x7eff580b5350 auth_method 0 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.096+0000 7eff577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.096+0000 7eff577fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.096+0000 7eff577fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.096+0000 7eff577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.096+0000 7eff577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.096+0000 7eff577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.096+0000 7eff57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:45.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.096+0000 7eff57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:45.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.096+0000 7eff57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:45.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.098+0000 7eff577fe640 10 monclient(hunting): handle_auth_done global_id 4251 payload 274 2024-05-15T12:32:45.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.098+0000 7eff577fe640 1 -- 172.21.15.49:0/3643586597 >> v2:172.21.15.88:3300/0 conn(0x7eff58145b60 msgr2=0x7eff58145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:45.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.098+0000 7eff577fe640 1 --2- 172.21.15.49:0/3643586597 >> v2:172.21.15.88:3300/0 conn(0x7eff58145b60 0x7eff58145ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:45.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.098+0000 7eff577fe640 1 -- 172.21.15.49:0/3643586597 >> v1:172.21.15.88:6789/0 conn(0x7eff5813f8f0 legacy=0x7eff5813fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:45.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.098+0000 7eff577fe640 10 monclient: _finish_hunting 0 2024-05-15T12:32:45.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.098+0000 7eff577fe640 1 monclient: found mon.noname-a 2024-05-15T12:32:45.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.098+0000 7eff577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:45.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.098+0000 7eff577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:45.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.098+0000 7eff577fe640 1 -- 172.21.15.49:0/3643586597 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff580ab400 con 0x7eff580b5350 2024-05-15T12:32:45.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.100+0000 7eff577fe640 1 --2- 172.21.15.49:0/3643586597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff580b5350 0x7eff580a9830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7eff580b28c0 tx=0x7eff4802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c32399c5a001b50 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:45.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 1 -- 172.21.15.49:0/3643586597 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff480315a0 con 0x7eff580b5350 2024-05-15T12:32:45.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 10 monclient: dump: 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:45.094 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 10 monclient: _finish_auth 0 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 10 monclient: _check_auth_tickets 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff5dbb3640 5 monclient: authenticate success, global_id 4251 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 1 -- 172.21.15.49:0/3643586597 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff48037070 con 0x7eff580b5350 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff5dbb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff5dbb3640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff5dbb3640 4 set_mon_vals no callback set 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 1 -- 172.21.15.49:0/3643586597 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48031960 con 0x7eff580b5350 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.103+0000 7eff567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff567fc640 10 monclient: dump: 2024-05-15T12:32:45.095 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff5dbb3640 10 monclient: shutdown 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff5dbb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff5dbb3640 1 -- 172.21.15.49:0/3643586597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff580b5350 msgr2=0x7eff580a9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff5dbb3640 1 --2- 172.21.15.49:0/3643586597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff580b5350 0x7eff580a9830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7eff580b28c0 tx=0x7eff4802b840 comp rx=0 tx=0).stop 2024-05-15T12:32:45.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff5dbb3640 1 -- 172.21.15.49:0/3643586597 shutdown_connections 2024-05-15T12:32:45.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff5dbb3640 1 --2- 172.21.15.49:0/3643586597 >> v2:172.21.15.88:3300/0 conn(0x7eff58145b60 0x7eff58145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:45.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff5dbb3640 1 --2- 172.21.15.49:0/3643586597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff580b5350 0x7eff580a9830 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:45.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff5dbb3640 1 -- 172.21.15.49:0/3643586597 >> 172.21.15.49:0/3643586597 conn(0x7eff58066ed0 msgr2=0x7eff580672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:45.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff5dbb3640 1 -- 172.21.15.49:0/3643586597 shutdown_connections 2024-05-15T12:32:45.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.104+0000 7eff5dbb3640 1 -- 172.21.15.49:0/3643586597 wait complete. 2024-05-15T12:32:45.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.105+0000 7eff5dbb3640 10 monclient: build_initial_monmap 2024-05-15T12:32:45.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.105+0000 7eff5dbb3640 10 monclient: monmap: 2024-05-15T12:32:45.100 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:45.100 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:45.100 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:45.100 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:45.100 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 librados: starting msgr at 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 librados: starting objecter 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 Processor -- start 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 -- start start 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 librados: setting wanted keys 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 librados: calling monclient init 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient: init 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:45.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff580b5350 0x7eff580acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient(hunting): picked mon.noname-b con 0x7eff580b5350 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff5813f8f0 0x7eff580ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient(hunting): picked mon.noname-a con 0x7eff5813f8f0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff58145b60 0x7eff581d7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient(hunting): picked mon.noname-c con 0x7eff58145b60 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff580aa470 con 0x7eff5813f8f0 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5814ceb0 con 0x7eff580b5350 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff580ae9d0 con 0x7eff58145b60 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.106+0000 7eff5dbb3640 10 monclient(hunting): authenticate will time out at 1523.268352s 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff577fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff580b5350 0x7eff580acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff57fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff58145b60 0x7eff581d7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff56ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff5813f8f0 0x7eff580ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff577fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff580b5350 0x7eff580acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:34900/0 (socket says 172.21.15.49:34900) 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff577fe640 1 -- 172.21.15.49:0/2754045634 learned_addr learned my addr 172.21.15.49:0/2754045634 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff577fe640 10 monclient(hunting): get_auth_request con 0x7eff580b5350 auth_method 0 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff577fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff577fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff56ffd640 10 monclient(hunting): get_auth_request con 0x7eff5813f8f0 auth_method 0 2024-05-15T12:32:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff56ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff57fff640 10 monclient(hunting): get_auth_request con 0x7eff58145b60 auth_method 0 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff57fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.107+0000 7eff57fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.108+0000 7eff577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.108+0000 7eff577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.108+0000 7eff577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.108+0000 7eff56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.108+0000 7eff56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.108+0000 7eff56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:45.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.108+0000 7eff57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.108+0000 7eff57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.108+0000 7eff57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.110+0000 7eff577fe640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.110+0000 7eff577fe640 1 -- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff58145b60 msgr2=0x7eff581d7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.110+0000 7eff577fe640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff58145b60 0x7eff581d7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.110+0000 7eff577fe640 1 -- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff5813f8f0 msgr2=0x7eff580ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.110+0000 7eff577fe640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff5813f8f0 0x7eff580ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.110+0000 7eff577fe640 10 monclient: _finish_hunting 0 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.110+0000 7eff577fe640 1 monclient: found mon.noname-b 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.110+0000 7eff577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.110+0000 7eff577fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.110+0000 7eff577fe640 1 -- 172.21.15.49:0/2754045634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff580adda0 con 0x7eff580b5350 2024-05-15T12:32:45.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.113+0000 7eff577fe640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff580b5350 0x7eff580acb30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7eff48000c00 tx=0x7eff480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:45.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.113+0000 7eff3ffff640 10 client.?.objecter ms_handle_connect 0x7eff580b5350 2024-05-15T12:32:45.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.113+0000 7eff3ffff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:45.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 1 -- 172.21.15.49:0/2754045634 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48032630 con 0x7eff580b5350 2024-05-15T12:32:45.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:45.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:45.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 10 monclient: dump: 2024-05-15T12:32:45.107 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 10 monclient: _finish_auth 0 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff5dbb3640 5 monclient: authenticate success, global_id 4264 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 1 -- 172.21.15.49:0/2754045634 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff480327f0 con 0x7eff580b5350 2024-05-15T12:32:45.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff5dbb3640 10 monclient: _renew_subs 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 1 -- 172.21.15.49:0/2754045634 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48032be0 con 0x7eff580b5350 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff5dbb3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff5dbb3640 1 -- 172.21.15.49:0/2754045634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff580aec40 con 0x7eff580b5350 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff3ffff640 10 monclient: dump: 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:45.109 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff55ffb640 4 set_mon_vals no callback set 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff5dbb3640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff5dbb3640 10 monclient: _renew_subs 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff5dbb3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff5dbb3640 1 -- 172.21.15.49:0/2754045634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff581d83e0 con 0x7eff580b5350 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.116+0000 7eff5dbb3640 1 librados: init done 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.117+0000 7eff5dbb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.117+0000 7eff5dbb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.117+0000 7eff5dbb3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:45.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.117+0000 7eff5dbb3640 1 -- 172.21.15.49:0/2754045634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff580aee10 con 0x7eff580b5350 2024-05-15T12:32:45.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.135+0000 7eff3ffff640 1 -- 172.21.15.49:0/2754045634 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7eff4804e400 con 0x7eff580b5350 2024-05-15T12:32:45.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.135+0000 7eff3ffff640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7eff4406ce20 0x7eff4406f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:45.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.135+0000 7eff56ffd640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7eff4406ce20 0x7eff4406f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:45.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.135+0000 7eff56ffd640 10 monclient: get_auth_request con 0x7eff4406ce20 auth_method 0 2024-05-15T12:32:45.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.135+0000 7eff56ffd640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7eff4406ce20 0x7eff4406f2e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7eff580b8ef0 tx=0x7eff4c007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:45.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.136+0000 7eff3ffff640 10 client.4264.objecter ms_handle_connect 0x7eff4406ce20 2024-05-15T12:32:45.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.137+0000 7eff3ffff640 1 -- 172.21.15.49:0/2754045634 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4857+0+0 (secure 0 0 0) 0x7eff4802f070 con 0x7eff580b5350 2024-05-15T12:32:45.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.137+0000 7eff3ffff640 10 client.4264.objecter ms_dispatch 0x7eff58000b90 osd_map(22..22 src has 1..22) v4 2024-05-15T12:32:45.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.137+0000 7eff3ffff640 3 client.4264.objecter handle_osd_map got epochs [22,22] > 0 2024-05-15T12:32:45.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.137+0000 7eff3ffff640 3 client.4264.objecter handle_osd_map decoding full epoch 22 2024-05-15T12:32:45.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.137+0000 7eff3ffff640 20 client.4264.objecter dump_active .. 0 homeless 2024-05-15T12:32:45.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.368+0000 7eff3ffff640 1 -- 172.21.15.49:0/2754045634 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7eff48040070 con 0x7eff580b5350 2024-05-15T12:32:45.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.368+0000 7eff3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:45.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.368+0000 7eff3ffff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:45.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.562+0000 7eff5dbb3640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-15T12:32:45.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.562+0000 7eff5dbb3640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-15T12:32:45.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.562+0000 7eff5dbb3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:45.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.562+0000 7eff5dbb3640 1 -- 172.21.15.49:0/2754045634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7eff5804f0e0 con 0x7eff580b5350 2024-05-15T12:32:45.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.877+0000 7eff3ffff640 1 -- 172.21.15.49:0/2754045634 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v23) v1 ==== 109+0+0 (secure 0 0 0) 0x7eff4807a9d0 con 0x7eff580b5350 2024-05-15T12:32:45.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.877+0000 7eff3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-15T12:32:45.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.877+0000 7eff3ffff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-05-15T12:32:45.866 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'rbd' created 2024-05-15T12:32:45.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 10 librados: watch_flush enter 2024-05-15T12:32:45.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 10 librados: watch_flush exit 2024-05-15T12:32:45.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 20 client.4264.objecter shutdown clearing up homeless session... 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 10 client.4264.objecter successfully canceled tick 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 -- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7eff4406ce20 msgr2=0x7eff4406f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7eff4406ce20 0x7eff4406f2e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7eff580b8ef0 tx=0x7eff4c007e30 comp rx=0 tx=0).stop 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 10 monclient: shutdown 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 -- 172.21.15.49:0/2754045634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff580b5350 msgr2=0x7eff580acb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff580b5350 0x7eff580acb30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7eff48000c00 tx=0x7eff480046c0 comp rx=0 tx=0).stop 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 -- 172.21.15.49:0/2754045634 shutdown_connections 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff58145b60 0x7eff581d7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff5813f8f0 0x7eff580ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7eff4406ce20 0x7eff4406f2e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 --2- 172.21.15.49:0/2754045634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff580b5350 0x7eff580acb30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 -- 172.21.15.49:0/2754045634 >> 172.21.15.49:0/2754045634 conn(0x7eff58066ed0 msgr2=0x7eff5813bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:45.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 -- 172.21.15.49:0/2754045634 shutdown_connections 2024-05-15T12:32:45.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 -- 172.21.15.49:0/2754045634 wait complete. 2024-05-15T12:32:45.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:45.879+0000 7eff5dbb3640 1 librados: shutdown 2024-05-15T12:32:45.881 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-15T12:32:46.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.036+0000 7f006cf56640 10 monclient: get_monmap_and_config 2024-05-15T12:32:46.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.036+0000 7f006cf56640 10 monclient: build_initial_monmap 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.036+0000 7f006cf56640 10 monclient: monmap: 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:46.037171+0000 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:46.037171+0000 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:46.026 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:46.027 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:46.027 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:46.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.037+0000 7f006cf56640 1 Processor -- start 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.037+0000 7f006cf56640 1 -- start start 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.037+0000 7f006cf56640 10 monclient: init 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0068143460 0x7f00680726c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 10 monclient(hunting): picked mon.noname-c con 0x7f0068143460 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0068070f60 0x7f00680b1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 10 monclient(hunting): picked mon.noname-b con 0x7f0068070f60 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f006806fb00 0x7f006806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 10 monclient(hunting): picked mon.noname-a con 0x7f006806fb00 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006813cfc0 con 0x7f006806fb00 2024-05-15T12:32:46.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00680704c0 con 0x7f0068143460 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0068070b10 con 0x7f0068070f60 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.038+0000 7f006cf56640 10 monclient(hunting): authenticate will time out at 1524.199755s 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f0065d9b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0068070f60 0x7f00680b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f006659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0068143460 0x7f00680726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f006659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0068143460 0x7f00680726c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:34912/0 (socket says 172.21.15.49:34912) 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f006659c640 1 -- 172.21.15.49:0/3880092373 learned_addr learned my addr 172.21.15.49:0/3880092373 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f006659c640 10 monclient(hunting): get_auth_request con 0x7f0068143460 auth_method 0 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f006659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:46.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f006659c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:46.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f006659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:46.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f0065d9b640 10 monclient(hunting): get_auth_request con 0x7f0068070f60 auth_method 0 2024-05-15T12:32:46.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f0065d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:46.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f0065d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:46.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.039+0000 7f0065d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:46.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.040+0000 7f006659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:46.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.040+0000 7f006659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:46.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.040+0000 7f006659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:46.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.040+0000 7f0065d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:46.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.040+0000 7f0065d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:46.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.040+0000 7f0065d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:46.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.041+0000 7f006659c640 10 monclient(hunting): handle_auth_done global_id 4267 payload 274 2024-05-15T12:32:46.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.041+0000 7f006659c640 1 -- 172.21.15.49:0/3880092373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0068070f60 msgr2=0x7f00680b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:46.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.041+0000 7f006659c640 1 --2- 172.21.15.49:0/3880092373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0068070f60 0x7f00680b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:46.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.041+0000 7f006659c640 1 -- 172.21.15.49:0/3880092373 >> v2:172.21.15.49:3300/0 conn(0x7f006806fb00 msgr2=0x7f006806ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:46.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.041+0000 7f006659c640 1 --2- 172.21.15.49:0/3880092373 >> v2:172.21.15.49:3300/0 conn(0x7f006806fb00 0x7f006806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:46.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.041+0000 7f006659c640 10 monclient: _finish_hunting 0 2024-05-15T12:32:46.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.041+0000 7f006659c640 1 monclient: found mon.noname-c 2024-05-15T12:32:46.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.041+0000 7f006659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:46.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.041+0000 7f006659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:46.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.041+0000 7f006659c640 1 -- 172.21.15.49:0/3880092373 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0068072d70 con 0x7f0068143460 2024-05-15T12:32:46.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.045+0000 7f006659c640 1 --2- 172.21.15.49:0/3880092373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 0x7f00680726c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f006806de90 tx=0x7f005802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4b27d98d1e103e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:46.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 1 -- 172.21.15.49:0/3880092373 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005802e070 con 0x7f0068143460 2024-05-15T12:32:46.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:46.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:46.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 10 monclient: dump: 2024-05-15T12:32:46.038 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:46.038 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:46.038 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:46.038 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 10 monclient: _finish_auth 0 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 10 monclient: _check_auth_tickets 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 1 -- 172.21.15.49:0/3880092373 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0058037070 con 0x7f0068143460 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006cf56640 5 monclient: authenticate success, global_id 4267 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006cf56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:46.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006cf56640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006cf56640 4 set_mon_vals no callback set 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 1 -- 172.21.15.49:0/3880092373 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058031a00 con 0x7f0068143460 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.049+0000 7f006559a640 10 monclient: dump: 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:46.040 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:46.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 10 monclient: shutdown 2024-05-15T12:32:46.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:46.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 1 -- 172.21.15.49:0/3880092373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 msgr2=0x7f00680726c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:46.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 1 --2- 172.21.15.49:0/3880092373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 0x7f00680726c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f006806de90 tx=0x7f005802b880 comp rx=0 tx=0).stop 2024-05-15T12:32:46.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 1 -- 172.21.15.49:0/3880092373 shutdown_connections 2024-05-15T12:32:46.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 1 --2- 172.21.15.49:0/3880092373 >> v2:172.21.15.49:3300/0 conn(0x7f006806fb00 0x7f006806ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:46.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 1 --2- 172.21.15.49:0/3880092373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0068070f60 0x7f00680b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:46.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 1 --2- 172.21.15.49:0/3880092373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 0x7f00680726c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:46.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 1 -- 172.21.15.49:0/3880092373 >> 172.21.15.49:0/3880092373 conn(0x7f00680b4440 msgr2=0x7f00680b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 1 -- 172.21.15.49:0/3880092373 shutdown_connections 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.050+0000 7f006cf56640 1 -- 172.21.15.49:0/3880092373 wait complete. 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.051+0000 7f006cf56640 10 monclient: build_initial_monmap 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.051+0000 7f006cf56640 10 monclient: monmap: 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:46.045 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.051+0000 7f006cf56640 1 librados: starting msgr at 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.051+0000 7f006cf56640 1 librados: starting objecter 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.051+0000 7f006cf56640 1 Processor -- start 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.051+0000 7f006cf56640 1 -- start start 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.051+0000 7f006cf56640 1 librados: setting wanted keys 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.051+0000 7f006cf56640 1 librados: calling monclient init 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.051+0000 7f006cf56640 10 monclient: init 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f006806fb00 0x7f006813de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 10 monclient(hunting): picked mon.noname-c con 0x7f006806fb00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0068070f60 0x7f006813e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:46.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 10 monclient(hunting): picked mon.noname-a con 0x7f0068070f60 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 0x7f00681356a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 10 monclient(hunting): picked mon.noname-b con 0x7f0068143460 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00680af820 con 0x7f0068070f60 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0068142870 con 0x7f0068143460 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00680af9a0 con 0x7f006806fb00 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.052+0000 7f006cf56640 10 monclient(hunting): authenticate will time out at 1524.214007s 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.053+0000 7f0066d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 0x7f00681356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.053+0000 7f0066d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 0x7f00681356a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:34920/0 (socket says 172.21.15.49:34920) 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.053+0000 7f0066d9d640 1 -- 172.21.15.49:0/2587932874 learned_addr learned my addr 172.21.15.49:0/2587932874 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:46.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.053+0000 7f0066d9d640 10 monclient(hunting): get_auth_request con 0x7f0068143460 auth_method 0 2024-05-15T12:32:46.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.053+0000 7f0066d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:46.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.053+0000 7f0066d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:46.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.053+0000 7f0066d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:46.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.054+0000 7f0066d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:46.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.054+0000 7f0066d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:46.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.054+0000 7f0066d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:46.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.055+0000 7f0065d9b640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0068070f60 0x7f006813e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:46.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0066d9d640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0066d9d640 1 -- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f006806fb00 msgr2=0x7f006813de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0066d9d640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f006806fb00 0x7f006813de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0066d9d640 1 -- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0068070f60 msgr2=0x7f006813e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0066d9d640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0068070f60 0x7f006813e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0066d9d640 10 monclient: _finish_hunting 0 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0066d9d640 1 monclient: found mon.noname-b 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0066d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0066d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0066d9d640 1 -- 172.21.15.49:0/2587932874 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006813fcc0 con 0x7f0068143460 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0065d9b640 10 monclient: get_auth_request con 0x7f0068070f60 auth_method 0 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.056+0000 7f0065d9b640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0068070f60 0x7f006813e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.058+0000 7f0066d9d640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 0x7f00681356a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f006000b970 tx=0x7f006000be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.058+0000 7f0056ffd640 10 client.?.objecter ms_handle_connect 0x7f0068143460 2024-05-15T12:32:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.058+0000 7f0056ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 1 -- 172.21.15.49:0/2587932874 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060019070 con 0x7f0068143460 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 10 monclient: dump: 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:46.053 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 10 monclient: _finish_auth 0 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 1 -- 172.21.15.49:0/2587932874 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006000cb60 con 0x7f0068143460 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f006cf56640 5 monclient: authenticate success, global_id 4270 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f006cf56640 10 monclient: _renew_subs 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f006cf56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f006cf56640 1 -- 172.21.15.49:0/2587932874 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006813ffa0 con 0x7f0068143460 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 1 -- 172.21.15.49:0/2587932874 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060014440 con 0x7f0068143460 2024-05-15T12:32:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0064d99640 4 set_mon_vals no callback set 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f006cf56640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f0056ffd640 10 monclient: dump: 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:46.055 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:46.056 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:46.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f006cf56640 10 monclient: _renew_subs 2024-05-15T12:32:46.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f006cf56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:46.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f006cf56640 1 -- 172.21.15.49:0/2587932874 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006813f080 con 0x7f0068143460 2024-05-15T12:32:46.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.063+0000 7f006cf56640 1 librados: init done 2024-05-15T12:32:46.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.064+0000 7f006cf56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:46.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.064+0000 7f006cf56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:46.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.064+0000 7f006cf56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:46.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.064+0000 7f006cf56640 1 -- 172.21.15.49:0/2587932874 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006804f0e0 con 0x7f0068143460 2024-05-15T12:32:46.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.081+0000 7f0056ffd640 1 -- 172.21.15.49:0/2587932874 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f006001c070 con 0x7f0068143460 2024-05-15T12:32:46.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.082+0000 7f0056ffd640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f004806ce20 0x7f004806f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:46.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.082+0000 7f006659c640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f004806ce20 0x7f004806f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:46.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.082+0000 7f006659c640 10 monclient: get_auth_request con 0x7f004806ce20 auth_method 0 2024-05-15T12:32:46.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.082+0000 7f0056ffd640 10 client.4270.objecter ms_handle_connect 0x7f004806ce20 2024-05-15T12:32:46.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.082+0000 7f006659c640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f004806ce20 0x7f004806f2e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0058036000 tx=0x7f0058030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:46.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.084+0000 7f0056ffd640 1 -- 172.21.15.49:0/2587932874 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f0060056b80 con 0x7f0068143460 2024-05-15T12:32:46.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.084+0000 7f0056ffd640 10 client.4270.objecter ms_dispatch 0x7f0068000b90 osd_map(23..23 src has 1..23) v4 2024-05-15T12:32:46.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.084+0000 7f0056ffd640 3 client.4270.objecter handle_osd_map got epochs [23,23] > 0 2024-05-15T12:32:46.074 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.084+0000 7f0056ffd640 3 client.4270.objecter handle_osd_map decoding full epoch 23 2024-05-15T12:32:46.074 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.084+0000 7f0056ffd640 20 client.4270.objecter dump_active .. 0 homeless 2024-05-15T12:32:46.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.331+0000 7f0056ffd640 1 -- 172.21.15.49:0/2587932874 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0060023020 con 0x7f0068143460 2024-05-15T12:32:46.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.331+0000 7f0056ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:46.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.331+0000 7f0056ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:46.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.537+0000 7f006cf56640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-15T12:32:46.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.537+0000 7f006cf56640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-15T12:32:46.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.537+0000 7f006cf56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:46.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:46.537+0000 7f006cf56640 1 -- 172.21.15.49:0/2587932874 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f006806b0c0 con 0x7f0068143460 2024-05-15T12:32:47.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.180+0000 7f0056ffd640 1 -- 172.21.15.49:0/2587932874 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v24) v1 ==== 173+0+0 (secure 0 0 0) 0x7f00600591e0 con 0x7f0068143460 2024-05-15T12:32:47.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.180+0000 7f0056ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-15T12:32:47.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.180+0000 7f0056ffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-05-15T12:32:47.170 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-15T12:32:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 10 librados: watch_flush enter 2024-05-15T12:32:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 10 librados: watch_flush exit 2024-05-15T12:32:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 20 client.4270.objecter shutdown clearing up homeless session... 2024-05-15T12:32:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 10 client.4270.objecter successfully canceled tick 2024-05-15T12:32:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 1 -- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f004806ce20 msgr2=0x7f004806f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f004806ce20 0x7f004806f2e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0058036000 tx=0x7f0058030000 comp rx=0 tx=0).stop 2024-05-15T12:32:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 10 monclient: shutdown 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 1 -- 172.21.15.49:0/2587932874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 msgr2=0x7f00681356a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 0x7f00681356a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f006000b970 tx=0x7f006000be30 comp rx=0 tx=0).stop 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 1 -- 172.21.15.49:0/2587932874 shutdown_connections 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f004806ce20 0x7f004806f2e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0068143460 0x7f00681356a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0068070f60 0x7f006813e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.182+0000 7f006cf56640 1 --2- 172.21.15.49:0/2587932874 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f006806fb00 0x7f006813de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.183+0000 7f006cf56640 1 -- 172.21.15.49:0/2587932874 >> 172.21.15.49:0/2587932874 conn(0x7f00680b4440 msgr2=0x7f0068133ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.183+0000 7f006cf56640 1 -- 172.21.15.49:0/2587932874 shutdown_connections 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.183+0000 7f006cf56640 1 -- 172.21.15.49:0/2587932874 wait complete. 2024-05-15T12:32:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.183+0000 7f006cf56640 1 librados: shutdown 2024-05-15T12:32:47.183 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-15T12:32:47.183 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-15T12:32:47.184 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-15T12:32:47.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-15T12:32:47.184 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-15T12:32:47.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.300+0000 7fe38b059640 10 monclient: get_monmap_and_config 2024-05-15T12:32:47.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.300+0000 7fe38b059640 10 monclient: build_initial_monmap 2024-05-15T12:32:47.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.300+0000 7fe38b059640 10 monclient: monmap: 2024-05-15T12:32:47.289 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:47.289 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:47.289 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:47.300770+0000 2024-05-15T12:32:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:47.300770+0000 2024-05-15T12:32:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:47.290 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:47.290 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:47.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 1 Processor -- start 2024-05-15T12:32:47.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 1 -- start start 2024-05-15T12:32:47.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 10 monclient: init 2024-05-15T12:32:47.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:47.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-05-15T12:32:47.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3840b53e0 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:47.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe384142870 0x7fe384142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:47.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 10 monclient(hunting): picked mon.noname-b con 0x7fe384142870 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:47.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe3841431f0 0x7fe38413f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:47.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 10 monclient(hunting): picked mon.noname-c con 0x7fe3841431f0 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:47.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3840ab230 con 0x7fe3840b53e0 2024-05-15T12:32:47.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:47.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3840b57c0 con 0x7fe3841431f0 2024-05-15T12:32:47.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3840aa230 con 0x7fe384142870 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.301+0000 7fe38b059640 10 monclient(hunting): authenticate will time out at 1525.463324s 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.302+0000 7fe383fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe384142870 0x7fe384142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.302+0000 7fe3895f6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe3841431f0 0x7fe38413f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe383fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe384142870 0x7fe384142cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:45942/0 (socket says 172.21.15.49:45942) 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe383fff640 1 -- 172.21.15.49:0/949470860 learned_addr learned my addr 172.21.15.49:0/949470860 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe383fff640 10 monclient(hunting): get_auth_request con 0x7fe384142870 auth_method 0 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe383fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe383fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe383fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe383fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe383fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:47.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe383fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:47.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe3895f6640 10 monclient(hunting): get_auth_request con 0x7fe3841431f0 auth_method 0 2024-05-15T12:32:47.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe3895f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:47.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe3895f6640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:47.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.303+0000 7fe3895f6640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.304+0000 7fe3895f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.304+0000 7fe3895f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.304+0000 7fe3895f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.305+0000 7fe383fff640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.305+0000 7fe383fff640 1 -- 172.21.15.49:0/949470860 >> v2:172.21.15.88:3300/0 conn(0x7fe3841431f0 msgr2=0x7fe38413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.305+0000 7fe383fff640 1 --2- 172.21.15.49:0/949470860 >> v2:172.21.15.88:3300/0 conn(0x7fe3841431f0 0x7fe38413f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.305+0000 7fe383fff640 1 -- 172.21.15.49:0/949470860 >> v1:172.21.15.88:6789/0 conn(0x7fe3840b53e0 legacy=0x7fe3840a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.305+0000 7fe383fff640 10 monclient: _finish_hunting 0 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.305+0000 7fe383fff640 1 monclient: found mon.noname-b 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.305+0000 7fe383fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.305+0000 7fe383fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:47.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.305+0000 7fe383fff640 1 -- 172.21.15.49:0/949470860 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3840ab510 con 0x7fe384142870 2024-05-15T12:32:47.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.307+0000 7fe383fff640 1 --2- 172.21.15.49:0/949470860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe384142870 0x7fe384142cb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe3840aa3b0 tx=0x7fe37002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7942bb8eb34628f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:47.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 1 -- 172.21.15.49:0/949470860 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370035020 con 0x7fe384142870 2024-05-15T12:32:47.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:47.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 10 monclient: dump: 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 10 monclient: _finish_auth 0 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 10 monclient: _check_auth_tickets 2024-05-15T12:32:47.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 1 -- 172.21.15.49:0/949470860 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe370031640 con 0x7fe384142870 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 1 -- 172.21.15.49:0/949470860 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37002d040 con 0x7fe384142870 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe38b059640 5 monclient: authenticate success, global_id 4223 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe38b059640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe38b059640 4 set_mon_vals no callback set 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.310+0000 7fe3837fe640 10 monclient: dump: 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:47.302 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:47.302 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:47.302 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:47.302 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.311+0000 7fe38b059640 10 monclient: shutdown 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.311+0000 7fe38b059640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.311+0000 7fe38b059640 1 -- 172.21.15.49:0/949470860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe384142870 msgr2=0x7fe384142cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.311+0000 7fe38b059640 1 --2- 172.21.15.49:0/949470860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe384142870 0x7fe384142cb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe3840aa3b0 tx=0x7fe37002b8f0 comp rx=0 tx=0).stop 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.311+0000 7fe38b059640 1 -- 172.21.15.49:0/949470860 shutdown_connections 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.311+0000 7fe38b059640 1 --2- 172.21.15.49:0/949470860 >> v2:172.21.15.88:3300/0 conn(0x7fe3841431f0 0x7fe38413f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.311+0000 7fe38b059640 1 --2- 172.21.15.49:0/949470860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe384142870 0x7fe384142cb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.311+0000 7fe38b059640 1 -- 172.21.15.49:0/949470860 >> 172.21.15.49:0/949470860 conn(0x7fe384066ed0 msgr2=0x7fe3840672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.311+0000 7fe38b059640 1 -- 172.21.15.49:0/949470860 shutdown_connections 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.311+0000 7fe38b059640 1 -- 172.21.15.49:0/949470860 wait complete. 2024-05-15T12:32:47.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.312+0000 7fe38b059640 10 monclient: build_initial_monmap 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.312+0000 7fe38b059640 10 monclient: monmap: 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.312+0000 7fe38b059640 1 librados: starting msgr at 2024-05-15T12:32:47.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.312+0000 7fe38b059640 1 librados: starting objecter 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.312+0000 7fe38b059640 1 Processor -- start 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 1 -- start start 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 1 librados: setting wanted keys 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 1 librados: calling monclient init 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient: init 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3840b53e0 0x7fe3841db480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient(hunting): picked mon.noname-c con 0x7fe3840b53e0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe384142870 0x7fe3841db9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient(hunting): picked mon.noname-b con 0x7fe384142870 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3841431f0 0x7fe3840b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient(hunting): picked mon.noname-a con 0x7fe3841431f0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:47.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38413ff30 con 0x7fe3841431f0 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38413fdb0 con 0x7fe384142870 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3841400b0 con 0x7fe3840b53e0 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.313+0000 7fe38b059640 10 monclient(hunting): authenticate will time out at 1525.475236s 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe388df5640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3840b53e0 0x7fe3841db480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe3895f6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3841431f0 0x7fe3840b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe383fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe384142870 0x7fe3841db9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe388df5640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3840b53e0 0x7fe3841db480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:45958/0 (socket says 172.21.15.49:45958) 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe388df5640 1 -- 172.21.15.49:0/3760277066 learned_addr learned my addr 172.21.15.49:0/3760277066 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe388df5640 10 monclient(hunting): get_auth_request con 0x7fe3840b53e0 auth_method 0 2024-05-15T12:32:47.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe388df5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe388df5640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe388df5640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe383fff640 10 monclient(hunting): get_auth_request con 0x7fe384142870 auth_method 0 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe383fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe383fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.314+0000 7fe383fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe3895f6640 10 monclient(hunting): get_auth_request con 0x7fe3841431f0 auth_method 0 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe3895f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe3895f6640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe3895f6640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe388df5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe388df5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe388df5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe3895f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:47.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe3895f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:47.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe3895f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:47.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe383fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:47.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe383fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:47.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.315+0000 7fe383fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.318+0000 7fe388df5640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.318+0000 7fe388df5640 1 -- 172.21.15.49:0/3760277066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe384142870 msgr2=0x7fe3841db9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.318+0000 7fe388df5640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe384142870 0x7fe3841db9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.318+0000 7fe388df5640 1 -- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3841431f0 msgr2=0x7fe3840b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.318+0000 7fe388df5640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3841431f0 0x7fe3840b0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.318+0000 7fe388df5640 10 monclient: _finish_hunting 0 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.318+0000 7fe388df5640 1 monclient: found mon.noname-c 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.318+0000 7fe388df5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.318+0000 7fe388df5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.318+0000 7fe388df5640 1 -- 172.21.15.49:0/3760277066 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3841c9650 con 0x7fe3840b53e0 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.321+0000 7fe388df5640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3840b53e0 0x7fe3841db480 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe37800c480 tx=0x7fe37800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.321+0000 7fe3817fa640 10 client.?.objecter ms_handle_connect 0x7fe3840b53e0 2024-05-15T12:32:47.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.321+0000 7fe3817fa640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:47.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe3817fa640 1 -- 172.21.15.49:0/3760277066 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3780052d0 con 0x7fe3840b53e0 2024-05-15T12:32:47.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe3817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe3817fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe3817fa640 10 monclient: dump: 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:47.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe3817fa640 10 monclient: _finish_auth 0 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe3817fa640 10 monclient: _check_auth_tickets 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe3817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe38b059640 5 monclient: authenticate success, global_id 4226 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe38b059640 10 monclient: _renew_subs 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe38b059640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe38b059640 1 -- 172.21.15.49:0/3760277066 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3841dbf00 con 0x7fe3840b53e0 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe38b059640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe38b059640 10 monclient: _renew_subs 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe38b059640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe38b059640 1 -- 172.21.15.49:0/3760277066 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3840b1300 con 0x7fe3840b53e0 2024-05-15T12:32:47.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.323+0000 7fe38b059640 1 librados: init done 2024-05-15T12:32:47.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.324+0000 7fe38b059640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:47.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.324+0000 7fe38b059640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:47.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.324+0000 7fe38b059640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:47.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.324+0000 7fe38b059640 1 -- 172.21.15.49:0/3760277066 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38404f0e0 con 0x7fe3840b53e0 2024-05-15T12:32:47.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.325+0000 7fe3817fa640 1 -- 172.21.15.49:0/3760277066 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe378005470 con 0x7fe3840b53e0 2024-05-15T12:32:47.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.325+0000 7fe3817fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.325+0000 7fe3817fa640 1 -- 172.21.15.49:0/3760277066 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378012830 con 0x7fe3840b53e0 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.325+0000 7fe3817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.325+0000 7fe3817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.325+0000 7fe3817fa640 10 monclient: dump: 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:47.316 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:47.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.325+0000 7fe382ffd640 4 set_mon_vals no callback set 2024-05-15T12:32:47.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.604+0000 7fe3817fa640 1 -- 172.21.15.49:0/3760277066 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fe3780129d0 con 0x7fe3840b53e0 2024-05-15T12:32:47.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.604+0000 7fe3817fa640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe36806cdd0 0x7fe36806f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:47.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.605+0000 7fe383fff640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe36806cdd0 0x7fe36806f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:47.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.605+0000 7fe383fff640 10 monclient: get_auth_request con 0x7fe36806cdd0 auth_method 0 2024-05-15T12:32:47.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.605+0000 7fe3817fa640 10 client.4226.objecter ms_handle_connect 0x7fe36806cdd0 2024-05-15T12:32:47.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.605+0000 7fe383fff640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe36806cdd0 0x7fe36806f290 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe370036000 tx=0x7fe37003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:47.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.618+0000 7fe3817fa640 1 -- 172.21.15.49:0/3760277066 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe378088d90 con 0x7fe3840b53e0 2024-05-15T12:32:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.618+0000 7fe3817fa640 10 client.4226.objecter ms_dispatch 0x7fe384000b90 osd_map(24..24 src has 1..24) v4 2024-05-15T12:32:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.618+0000 7fe3817fa640 3 client.4226.objecter handle_osd_map got epochs [24,24] > 0 2024-05-15T12:32:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.618+0000 7fe3817fa640 3 client.4226.objecter handle_osd_map decoding full epoch 24 2024-05-15T12:32:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.618+0000 7fe3817fa640 20 client.4226.objecter dump_active .. 0 homeless 2024-05-15T12:32:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.619+0000 7fe3817fa640 1 -- 172.21.15.49:0/3760277066 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe378059060 con 0x7fe3840b53e0 2024-05-15T12:32:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.619+0000 7fe3817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.619+0000 7fe3817fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:47.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.867+0000 7fe38b059640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-05-15T12:32:47.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.867+0000 7fe38b059640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-05-15T12:32:47.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.867+0000 7fe38b059640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:47.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:47.867+0000 7fe38b059640 1 -- 172.21.15.49:0/3760277066 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe38413efa0 con 0x7fe3840b53e0 2024-05-15T12:32:48.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.025+0000 7fe3817fa640 1 -- 172.21.15.49:0/3760277066 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+171689 (secure 0 0 0) 0x7fe37805d100 con 0x7fe3840b53e0 2024-05-15T12:32:48.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.025+0000 7fe3817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-05-15T12:32:48.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.025+0000 7fe3817fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:32:48.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.028+0000 7fe38b059640 10 librados: watch_flush enter 2024-05-15T12:32:48.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.028+0000 7fe38b059640 10 librados: watch_flush exit 2024-05-15T12:32:48.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.028+0000 7fe38b059640 20 client.4226.objecter shutdown clearing up homeless session... 2024-05-15T12:32:48.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.028+0000 7fe38b059640 10 client.4226.objecter successfully canceled tick 2024-05-15T12:32:48.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 1 -- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe36806cdd0 msgr2=0x7fe36806f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe36806cdd0 0x7fe36806f290 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe370036000 tx=0x7fe37003c040 comp rx=0 tx=0).stop 2024-05-15T12:32:48.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 10 monclient: shutdown 2024-05-15T12:32:48.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:48.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 1 -- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3840b53e0 msgr2=0x7fe3841db480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3840b53e0 0x7fe3841db480 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe37800c480 tx=0x7fe37800c940 comp rx=0 tx=0).stop 2024-05-15T12:32:48.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 1 -- 172.21.15.49:0/3760277066 shutdown_connections 2024-05-15T12:32:48.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3841431f0 0x7fe3840b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe36806cdd0 0x7fe36806f290 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe384142870 0x7fe3841db9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 1 --2- 172.21.15.49:0/3760277066 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3840b53e0 0x7fe3841db480 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.029+0000 7fe38b059640 1 -- 172.21.15.49:0/3760277066 >> 172.21.15.49:0/3760277066 conn(0x7fe384066ed0 msgr2=0x7fe38413dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:48.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.030+0000 7fe38b059640 1 -- 172.21.15.49:0/3760277066 shutdown_connections 2024-05-15T12:32:48.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.030+0000 7fe38b059640 1 -- 172.21.15.49:0/3760277066 wait complete. 2024-05-15T12:32:48.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.030+0000 7fe38b059640 1 librados: shutdown 2024-05-15T12:32:48.033 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":4,"flags":0,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":3444323000},{"type":"v1","addr":"172.21.15.49:6801","nonce":3444323000}]},"active_addr":"172.21.15.49:6801/3444323000","active_change":"2024-05-15T12:31:48.061450+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":4291558882}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":546102664}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":3804161322}]}]} 2024-05-15T12:32:48.039 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-15T12:32:48.039 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-15T12:32:48.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-15T12:32:48.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.161+0000 7fd5a677e640 10 monclient: get_monmap_and_config 2024-05-15T12:32:48.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.161+0000 7fd5a677e640 10 monclient: build_initial_monmap 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.161+0000 7fd5a677e640 10 monclient: monmap: 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:48.162174+0000 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:48.162174+0000 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:48.151 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.162+0000 7fd5a677e640 1 Processor -- start 2024-05-15T12:32:48.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.162+0000 7fd5a677e640 1 -- start start 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.162+0000 7fd5a677e640 10 monclient: init 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5a00b53e0 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a0142870 0x7fd5a0142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a0142870 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5a01431f0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a00ab230 con 0x7fd5a01431f0 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a00ab510 con 0x7fd5a00b53e0 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00b57c0 con 0x7fd5a0142870 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.163+0000 7fd5a677e640 10 monclient(hunting): authenticate will time out at 1526.324723s 2024-05-15T12:32:48.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.165+0000 7fd59ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd5a00b53e0 0x7fd5a00a99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:36874/0 (socket says 172.21.15.49:36874) 2024-05-15T12:32:48.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.165+0000 7fd59ffff640 1 -- 172.21.15.49:0/3539828020 learned_addr learned my addr 172.21.15.49:0/3539828020 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:48.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.206+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067661739 0 0) 0x7fd5a00ab510 con 0x7fd5a00b53e0 2024-05-15T12:32:48.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.206+0000 7fd59effd640 10 monclient(hunting): _negotiate 2024-05-15T12:32:48.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.206+0000 7fd59effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:48.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.206+0000 7fd59effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:48.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.206+0000 7fd59effd640 10 monclient(hunting): my global_id is 4279 2024-05-15T12:32:48.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.206+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd588003520 con 0x7fd5a00b53e0 2024-05-15T12:32:48.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.206+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214357116 0 0) 0x7fd5a00ab230 con 0x7fd5a01431f0 2024-05-15T12:32:48.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.207+0000 7fd59effd640 10 monclient(hunting): _negotiate 2024-05-15T12:32:48.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.207+0000 7fd59effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:48.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.207+0000 7fd59effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:48.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.207+0000 7fd59effd640 10 monclient(hunting): my global_id is 4260 2024-05-15T12:32:48.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.207+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a00ab510 con 0x7fd5a01431f0 2024-05-15T12:32:48.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.210+0000 7fd59f7fe640 1 --2- 172.21.15.49:0/3539828020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a0142870 0x7fd5a0142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.212+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 <== mon.0 v1:172.21.15.49:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1348514905 0 0) 0x7fd5a00ab510 con 0x7fd5a01431f0 2024-05-15T12:32:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.212+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a0142870 msgr2=0x7fd5a0142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.212+0000 7fd59effd640 1 --2- 172.21.15.49:0/3539828020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a0142870 0x7fd5a0142cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.212+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 >> v1:172.21.15.88:6789/0 conn(0x7fd5a00b53e0 legacy=0x7fd5a00a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.212+0000 7fd59effd640 10 monclient: _finish_hunting 0 2024-05-15T12:32:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.212+0000 7fd59effd640 1 monclient: found mon.noname-a-legacy 2024-05-15T12:32:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.212+0000 7fd59effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.212+0000 7fd59effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.49:6789/0 2024-05-15T12:32:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.212+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 --> v1:172.21.15.49:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a013fd70 con 0x7fd5a01431f0 2024-05-15T12:32:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient: _finish_auth 0 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 <== mon.0 v1:172.21.15.49:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2686122088 0 0) 0x7fd5980019f0 con 0x7fd5a01431f0 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd5a677e640 5 monclient: authenticate success, global_id 4260 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd5a677e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient: dump: 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] but i'm connected to v1:172.21.15.49:6789/0, reconnecting 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 >> v1:172.21.15.49:6789/0 conn(0x7fd5a01431f0 legacy=0x7fd5a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 1 --2- 172.21.15.49:0/3539828020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd588003be0 0x7fd588003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient(hunting): picked mon.b con 0x7fd588003be0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 1 --2- 172.21.15.49:0/3539828020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd588004800 0x7fd588004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient(hunting): picked mon.a con 0x7fd588004800 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 1 --2- 172.21.15.49:0/3539828020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a0142870 0x7fd58800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient(hunting): picked mon.c con 0x7fd5a0142870 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:48.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00b57c0 con 0x7fd588004800 2024-05-15T12:32:48.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd588005620 con 0x7fd588003be0 2024-05-15T12:32:48.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58800bf30 con 0x7fd5a0142870 2024-05-15T12:32:48.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:48.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd59effd640 10 monclient(hunting): _finish_auth 0 2024-05-15T12:32:48.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.213+0000 7fd5a677e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:48.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.214+0000 7fd59ffff640 1 --2- 172.21.15.49:0/3539828020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd588003be0 0x7fd588003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:48.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.215+0000 7fd59ffff640 10 monclient(hunting): get_auth_request con 0x7fd588003be0 auth_method 0 2024-05-15T12:32:48.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.215+0000 7fd59ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:48.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.215+0000 7fd59ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:48.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.215+0000 7fd59ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.215+0000 7fd59ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.215+0000 7fd59ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.215+0000 7fd59ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 10 monclient(hunting): handle_auth_done global_id 4260 payload 293 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 1 -- 172.21.15.49:0/3539828020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a0142870 msgr2=0x7fd58800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 1 --2- 172.21.15.49:0/3539828020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a0142870 0x7fd58800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 1 -- 172.21.15.49:0/3539828020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd588004800 msgr2=0x7fd588004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 1 --2- 172.21.15.49:0/3539828020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd588004800 0x7fd588004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 10 monclient: _finish_hunting 0 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 1 monclient: found mon.b 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 1 -- 172.21.15.49:0/3539828020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a00ab510 con 0x7fd588003be0 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 10 monclient: _finish_auth 0 2024-05-15T12:32:48.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:48.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.217+0000 7fd59ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:48.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.219+0000 7fd59ffff640 1 --2- 172.21.15.49:0/3539828020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd588003be0 0x7fd588003fe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5900019c0 tx=0x7fd588005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16169057ee6319bf server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:48.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590036070 con 0x7fd588003be0 2024-05-15T12:32:48.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd59effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:48.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd59effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:48.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd59effd640 10 monclient: dump: 2024-05-15T12:32:48.212 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:48.212 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.212 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.212 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.212 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5900048a0 con 0x7fd588003be0 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd5a677e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd59effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd5a677e640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd5a677e640 4 set_mon_vals no callback set 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd59effd640 1 -- 172.21.15.49:0/3539828020 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590001610 con 0x7fd588003be0 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd59effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd59effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:48.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.223+0000 7fd59effd640 10 monclient: dump: 2024-05-15T12:32:48.214 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:48.214 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.214 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.214 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.214 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:48.214 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.214 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:48.214 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:48.214 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:48.214 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.224+0000 7fd5a677e640 10 monclient: shutdown 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.224+0000 7fd5a677e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.224+0000 7fd5a677e640 1 -- 172.21.15.49:0/3539828020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd588003be0 msgr2=0x7fd588003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.224+0000 7fd5a677e640 1 --2- 172.21.15.49:0/3539828020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd588003be0 0x7fd588003fe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5900019c0 tx=0x7fd588005830 comp rx=0 tx=0).stop 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.224+0000 7fd5a4d1b640 1 -- 172.21.15.49:0/3539828020 reap_dead start 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.224+0000 7fd5a677e640 1 -- 172.21.15.49:0/3539828020 shutdown_connections 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.224+0000 7fd5a677e640 1 -- 172.21.15.49:0/3539828020 >> 172.21.15.49:0/3539828020 conn(0x7fd5a0066ed0 msgr2=0x7fd5a00672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.224+0000 7fd5a677e640 1 -- 172.21.15.49:0/3539828020 shutdown_connections 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.224+0000 7fd5a677e640 1 -- 172.21.15.49:0/3539828020 wait complete. 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.225+0000 7fd5a677e640 10 monclient: build_initial_monmap 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.225+0000 7fd5a677e640 10 monclient: monmap: 2024-05-15T12:32:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.225+0000 7fd5a677e640 1 librados: starting msgr at 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.225+0000 7fd5a677e640 1 librados: starting objecter 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.225+0000 7fd5a677e640 1 Processor -- start 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.225+0000 7fd5a677e640 1 -- start start 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.225+0000 7fd5a677e640 1 librados: setting wanted keys 2024-05-15T12:32:48.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.225+0000 7fd5a677e640 1 librados: calling monclient init 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.225+0000 7fd5a677e640 10 monclient: init 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd588003be0 0x7fd5a01db2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 10 monclient(hunting): picked mon.noname-b con 0x7fd588003be0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5a01dbcb0 0x7fd5a00ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 10 monclient(hunting): picked mon.noname-a con 0x7fd5a01dbcb0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a00ad730 0x7fd5a0139270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a00ad730 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00aa230 con 0x7fd5a01dbcb0 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0142f40 con 0x7fd588003be0 2024-05-15T12:32:48.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0140000 con 0x7fd5a00ad730 2024-05-15T12:32:48.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:48.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.226+0000 7fd5a677e640 10 monclient(hunting): authenticate will time out at 1526.388361s 2024-05-15T12:32:48.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.227+0000 7fd5a4d1b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a00ad730 0x7fd5a0139270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:48.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.227+0000 7fd5a4d1b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a00ad730 0x7fd5a0139270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:45976/0 (socket says 172.21.15.49:45976) 2024-05-15T12:32:48.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.227+0000 7fd5a4d1b640 1 -- 172.21.15.49:0/120035497 learned_addr learned my addr 172.21.15.49:0/120035497 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:48.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.227+0000 7fd5a4d1b640 10 monclient(hunting): get_auth_request con 0x7fd5a00ad730 auth_method 0 2024-05-15T12:32:48.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.227+0000 7fd5a4d1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:48.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.227+0000 7fd5a4d1b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:48.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.227+0000 7fd5a4d1b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:48.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.228+0000 7fd5a4d1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:48.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.228+0000 7fd5a4d1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:48.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.228+0000 7fd5a4d1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:48.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.228+0000 7fd59ffff640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd588003be0 0x7fd5a01db2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:48.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.232+0000 7fd5a4d1b640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-05-15T12:32:48.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.232+0000 7fd5a4d1b640 1 -- 172.21.15.49:0/120035497 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd588003be0 msgr2=0x7fd5a01db2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.232+0000 7fd5a4d1b640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd588003be0 0x7fd5a01db2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.232+0000 7fd5a4d1b640 1 -- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5a01dbcb0 msgr2=0x7fd5a00ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.232+0000 7fd5a4d1b640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5a01dbcb0 0x7fd5a00ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.232+0000 7fd5a4d1b640 10 monclient: _finish_hunting 0 2024-05-15T12:32:48.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.232+0000 7fd5a4d1b640 1 monclient: found mon.noname-c 2024-05-15T12:32:48.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.232+0000 7fd5a4d1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:48.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.232+0000 7fd5a4d1b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:48.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.232+0000 7fd5a4d1b640 1 -- 172.21.15.49:0/120035497 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a01db800 con 0x7fd5a00ad730 2024-05-15T12:32:48.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.236+0000 7fd5a4d1b640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a00ad730 0x7fd5a0139270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd5a01435d0 tx=0x7fd59800ccd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:48.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.236+0000 7fd59cff9640 10 client.?.objecter ms_handle_connect 0x7fd5a00ad730 2024-05-15T12:32:48.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.236+0000 7fd59cff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:48.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59cff9640 1 -- 172.21.15.49:0/120035497 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598017070 con 0x7fd5a00ad730 2024-05-15T12:32:48.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:48.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:48.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59cff9640 10 monclient: dump: 2024-05-15T12:32:48.295 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:48.295 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.295 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.295 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.295 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:48.295 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59cff9640 10 monclient: _finish_auth 0 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59cff9640 10 monclient: _check_auth_tickets 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59cff9640 1 -- 172.21.15.49:0/120035497 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5980070d0 con 0x7fd5a00ad730 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59cff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59e7fc640 4 set_mon_vals no callback set 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd59cff9640 1 -- 172.21.15.49:0/120035497 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5980074a0 con 0x7fd5a00ad730 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd5a677e640 5 monclient: authenticate success, global_id 4229 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd5a677e640 10 monclient: _renew_subs 2024-05-15T12:32:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd5a677e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.305+0000 7fd5a677e640 1 -- 172.21.15.49:0/120035497 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a00b1c40 con 0x7fd5a00ad730 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.306+0000 7fd59cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.306+0000 7fd59cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.306+0000 7fd59cff9640 10 monclient: dump: 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:48.297 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.306+0000 7fd5a677e640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:48.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.306+0000 7fd5a677e640 10 monclient: _renew_subs 2024-05-15T12:32:48.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.306+0000 7fd5a677e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:48.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.306+0000 7fd5a677e640 1 -- 172.21.15.49:0/120035497 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a01397b0 con 0x7fd5a00ad730 2024-05-15T12:32:48.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.306+0000 7fd5a677e640 1 librados: init done 2024-05-15T12:32:48.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.307+0000 7fd5a677e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:48.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.307+0000 7fd5a677e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:48.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.307+0000 7fd5a677e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:48.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.307+0000 7fd5a677e640 1 -- 172.21.15.49:0/120035497 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a004f0e0 con 0x7fd5a00ad730 2024-05-15T12:32:48.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.591+0000 7fd59cff9640 1 -- 172.21.15.49:0/120035497 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fd59801a070 con 0x7fd5a00ad730 2024-05-15T12:32:48.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.592+0000 7fd59cff9640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd58c06cdd0 0x7fd58c06f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.592+0000 7fd59ffff640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd58c06cdd0 0x7fd58c06f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:48.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.592+0000 7fd59ffff640 10 monclient: get_auth_request con 0x7fd58c06cdd0 auth_method 0 2024-05-15T12:32:48.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.592+0000 7fd59cff9640 10 client.4229.objecter ms_handle_connect 0x7fd58c06cdd0 2024-05-15T12:32:48.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.592+0000 7fd59ffff640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd58c06cdd0 0x7fd58c06f290 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd590033000 tx=0x7fd590009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:48.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.606+0000 7fd59cff9640 1 -- 172.21.15.49:0/120035497 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd598056470 con 0x7fd5a00ad730 2024-05-15T12:32:48.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.606+0000 7fd59cff9640 10 client.4229.objecter ms_dispatch 0x7fd5a0000b90 osd_map(25..25 src has 1..25) v4 2024-05-15T12:32:48.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.606+0000 7fd59cff9640 3 client.4229.objecter handle_osd_map got epochs [25,25] > 0 2024-05-15T12:32:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.606+0000 7fd59cff9640 3 client.4229.objecter handle_osd_map decoding full epoch 25 2024-05-15T12:32:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.606+0000 7fd59cff9640 20 client.4229.objecter dump_active .. 0 homeless 2024-05-15T12:32:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.606+0000 7fd59cff9640 1 -- 172.21.15.49:0/120035497 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd5980b58c0 con 0x7fd5a00ad730 2024-05-15T12:32:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.606+0000 7fd59cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.606+0000 7fd59cff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:48.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.793+0000 7fd5a677e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:48.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.793+0000 7fd5a677e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:48.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.793+0000 7fd5a677e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:48.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.793+0000 7fd5a677e640 1 -- 172.21.15.49:0/120035497 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5a00716f0 con 0x7fd5a00ad730 2024-05-15T12:32:48.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.845+0000 7fd59cff9640 1 -- 172.21.15.49:0/120035497 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+14694 (secure 0 0 0) 0x7fd598021020 con 0x7fd5a00ad730 2024-05-15T12:32:48.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.845+0000 7fd59cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:48.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.845+0000 7fd59cff9640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:48.834 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:32:48.835 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":25,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:32:48.095674+0000","last_up_change":"2024-05-15T12:32:41.022082+0000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:32:30.595515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-15T12:32:45.576288+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6819","nonce":3376151072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6821","nonce":3376151072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6825","nonce":3376151072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6823","nonce":3376151072}]},"public_addr":"172.21.15.49:6819/3376151072","cluster_addr":"172.21.15.49:6821/3376151072","heartbeat_back_addr":"172.21.15.49:6825/3376151072","heartbeat_front_addr":"172.21.15.49:6823/3376151072","state":["exists","up"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6811","nonce":3250791401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6813","nonce":3250791401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6817","nonce":3250791401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6815","nonce":3250791401}]},"public_addr":"172.21.15.49:6811/3250791401","cluster_addr":"172.21.15.49:6813/3250791401","heartbeat_back_addr":"172.21.15.49:6817/3250791401","heartbeat_front_addr":"172.21.15.49:6815/3250791401","state":["exists","up"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6827","nonce":2361777220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6829","nonce":2361777220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6833","nonce":2361777220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6831","nonce":2361777220}]},"public_addr":"172.21.15.49:6827/2361777220","cluster_addr":"172.21.15.49:6829/2361777220","heartbeat_back_addr":"172.21.15.49:6833/2361777220","heartbeat_front_addr":"172.21.15.49:6831/2361777220","state":["exists","up"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6803","nonce":945312262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6805","nonce":945312262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6809","nonce":945312262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6807","nonce":945312262}]},"public_addr":"172.21.15.49:6803/945312262","cluster_addr":"172.21.15.49:6805/945312262","heartbeat_back_addr":"172.21.15.49:6809/945312262","heartbeat_front_addr":"172.21.15.49:6807/945312262","state":["exists","up"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6817","nonce":3384996234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6819","nonce":3384996234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6823","nonce":3384996234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6821","nonce":3384996234}]},"public_addr":"172.21.15.88:6817/3384996234","cluster_addr":"172.21.15.88:6819/3384996234","heartbeat_back_addr":"172.21.15.88:6823/3384996234","heartbeat_front_addr":"172.21.15.88:6821/3384996234","state":["exists","up"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6809","nonce":3085185190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6811","nonce":3085185190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6815","nonce":3085185190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6813","nonce":3085185190}]},"public_addr":"172.21.15.88:6809/3085185190","cluster_addr":"172.21.15.88:6811/3085185190","heartbeat_back_addr":"172.21.15.88:6815/3085185190","heartbeat_front_addr":"172.21.15.88:6813/3085185190","state":["exists","up"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6801","nonce":3138490191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6803","nonce":3138490191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6807","nonce":3138490191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6805","nonce":3138490191}]},"public_addr":"172.21.15.88:6801/3138490191","cluster_addr":"172.21.15.88:6803/3138490191","heartbeat_back_addr":"172.21.15.88:6807/3138490191","heartbeat_front_addr":"172.21.15.88:6805/3138490191","state":["exists","up"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6825","nonce":2037618931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6827","nonce":2037618931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6831","nonce":2037618931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6829","nonce":2037618931}]},"public_addr":"172.21.15.88:6825/2037618931","cluster_addr":"172.21.15.88:6827/2037618931","heartbeat_back_addr":"172.21.15.88:6831/2037618931","heartbeat_front_addr":"172.21.15.88:6829/2037618931","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:28.806234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:29.001705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:28.402190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:27.582000+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:25.805322+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.269314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.001531+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.945037+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 10 librados: watch_flush enter 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 10 librados: watch_flush exit 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 20 client.4229.objecter shutdown clearing up homeless session... 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 10 client.4229.objecter successfully canceled tick 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 -- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd58c06cdd0 msgr2=0x7fd58c06f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd58c06cdd0 0x7fd58c06f290 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd590033000 tx=0x7fd590009030 comp rx=0 tx=0).stop 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 10 monclient: shutdown 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 -- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a00ad730 msgr2=0x7fd5a0139270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a00ad730 0x7fd5a0139270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd5a01435d0 tx=0x7fd59800ccd0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 -- 172.21.15.49:0/120035497 shutdown_connections 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5a00ad730 0x7fd5a0139270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5a01dbcb0 0x7fd5a00ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd58c06cdd0 0x7fd58c06f290 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 --2- 172.21.15.49:0/120035497 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd588003be0 0x7fd5a01db2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 -- 172.21.15.49:0/120035497 >> 172.21.15.49:0/120035497 conn(0x7fd5a0066ed0 msgr2=0x7fd5a00672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:48.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 -- 172.21.15.49:0/120035497 shutdown_connections 2024-05-15T12:32:48.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 -- 172.21.15.49:0/120035497 wait complete. 2024-05-15T12:32:48.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.847+0000 7fd5a677e640 1 librados: shutdown 2024-05-15T12:32:48.848 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-15T12:32:48.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-15T12:32:48.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.963+0000 7f61e799a640 10 monclient: get_monmap_and_config 2024-05-15T12:32:48.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.963+0000 7f61e799a640 10 monclient: build_initial_monmap 2024-05-15T12:32:48.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.963+0000 7f61e799a640 10 monclient: monmap: 2024-05-15T12:32:48.952 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:48.964065+0000 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:48.964065+0000 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:48.953 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.964+0000 7f61e799a640 1 Processor -- start 2024-05-15T12:32:48.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.964+0000 7f61e799a640 1 -- start start 2024-05-15T12:32:48.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.964+0000 7f61e799a640 10 monclient: init 2024-05-15T12:32:48.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.964+0000 7f61e799a640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.964+0000 7f61e799a640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.964+0000 7f61e799a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e00a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.964+0000 7f61e799a640 10 monclient(hunting): picked mon.noname-b con 0x7f61e00b5470 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.964+0000 7f61e799a640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f61e00aa630 0x7f61e0145c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.964+0000 7f61e799a640 10 monclient(hunting): picked mon.noname-a con 0x7f61e00aa630 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.964+0000 7f61e799a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61e01461c0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.965+0000 7f61e799a640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e00ab8d0 con 0x7f61e01461c0 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.965+0000 7f61e799a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.965+0000 7f61e799a640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e00b5850 con 0x7f61e00aa630 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.965+0000 7f61e799a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.965+0000 7f61e799a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00aa1a0 con 0x7f61e00b5470 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.965+0000 7f61e799a640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.965+0000 7f61e799a640 10 monclient(hunting): authenticate will time out at 1527.126474s 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.965+0000 7f61e4f35640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f61e00aa630 0x7f61e0145c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.965+0000 7f61e5736640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e00a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e5736640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e00a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:45980/0 (socket says 172.21.15.49:45980) 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e5736640 1 -- 172.21.15.49:0/1282563098 learned_addr learned my addr 172.21.15.49:0/1282563098 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e5736640 10 monclient(hunting): get_auth_request con 0x7f61e00b5470 auth_method 0 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e5736640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e5736640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e5736640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e4f35640 10 monclient(hunting): get_auth_request con 0x7f61e00aa630 auth_method 0 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e4f35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e4f35640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e4f35640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e5736640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e5736640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e5736640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:48.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e4f35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:48.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e4f35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:48.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.966+0000 7f61e4f35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:48.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.968+0000 7f61e5736640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-05-15T12:32:48.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.968+0000 7f61e5736640 1 -- 172.21.15.49:0/1282563098 >> v2:172.21.15.49:3300/0 conn(0x7f61e00aa630 msgr2=0x7f61e0145c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:48.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.968+0000 7f61e5736640 1 --2- 172.21.15.49:0/1282563098 >> v2:172.21.15.49:3300/0 conn(0x7f61e00aa630 0x7f61e0145c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.968+0000 7f61e5736640 1 -- 172.21.15.49:0/1282563098 >> v1:172.21.15.49:6789/0 conn(0x7f61e01461c0 legacy=0x7f61e013f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.968+0000 7f61e5736640 10 monclient: _finish_hunting 0 2024-05-15T12:32:48.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.968+0000 7f61e5736640 1 monclient: found mon.noname-b 2024-05-15T12:32:48.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.968+0000 7f61e5736640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:48.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.968+0000 7f61e5736640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:48.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.968+0000 7f61e5736640 1 -- 172.21.15.49:0/1282563098 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e00a9ea0 con 0x7f61e00b5470 2024-05-15T12:32:48.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.970+0000 7f61e5736640 1 --2- 172.21.15.49:0/1282563098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e00a9960 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009870 tx=0x7f61dc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c55759e4be63529 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:48.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.973+0000 7f61d7fff640 1 -- 172.21.15.49:0/1282563098 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc0317c0 con 0x7f61e00b5470 2024-05-15T12:32:48.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.973+0000 7f61d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:48.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.973+0000 7f61d7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:48.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.973+0000 7f61d7fff640 10 monclient: dump: 2024-05-15T12:32:48.962 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:48.962 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.962 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.962 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.962 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:48.962 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.963 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:48.963 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:48.963 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:48.963 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.973+0000 7f61d7fff640 10 monclient: _finish_auth 0 2024-05-15T12:32:48.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.973+0000 7f61d7fff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:48.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.973+0000 7f61d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:48.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.973+0000 7f61e799a640 5 monclient: authenticate success, global_id 4232 2024-05-15T12:32:48.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.973+0000 7f61e799a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.974+0000 7f61d7fff640 1 -- 172.21.15.49:0/1282563098 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc037070 con 0x7f61e00b5470 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.974+0000 7f61d7fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.974+0000 7f61d7fff640 1 -- 172.21.15.49:0/1282563098 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc031b80 con 0x7f61e00b5470 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.974+0000 7f61e799a640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.974+0000 7f61e799a640 4 set_mon_vals no callback set 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.974+0000 7f61d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.974+0000 7f61d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.974+0000 7f61d7fff640 10 monclient: dump: 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.965 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 10 monclient: shutdown 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 1 -- 172.21.15.49:0/1282563098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 msgr2=0x7f61e00a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 1 --2- 172.21.15.49:0/1282563098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e00a9960 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009870 tx=0x7f61dc02b860 comp rx=0 tx=0).stop 2024-05-15T12:32:48.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 1 -- 172.21.15.49:0/1282563098 shutdown_connections 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 1 --2- 172.21.15.49:0/1282563098 >> v2:172.21.15.49:3300/0 conn(0x7f61e00aa630 0x7f61e0145c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 1 --2- 172.21.15.49:0/1282563098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e00a9960 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 1 -- 172.21.15.49:0/1282563098 >> 172.21.15.49:0/1282563098 conn(0x7f61e0066f00 msgr2=0x7f61e0067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 1 -- 172.21.15.49:0/1282563098 shutdown_connections 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 1 -- 172.21.15.49:0/1282563098 wait complete. 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 10 monclient: build_initial_monmap 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.975+0000 7f61e799a640 10 monclient: monmap: 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:48.967 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:48.968 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:48.968 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 1 librados: starting msgr at 2024-05-15T12:32:48.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 1 librados: starting objecter 2024-05-15T12:32:48.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 1 Processor -- start 2024-05-15T12:32:48.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 1 -- start start 2024-05-15T12:32:48.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 1 librados: setting wanted keys 2024-05-15T12:32:48.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 1 librados: calling monclient init 2024-05-15T12:32:48.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 10 monclient: init 2024-05-15T12:32:48.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:48.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:32:48.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61e00aa630 0x7f61e012dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 10 monclient(hunting): picked mon.noname-b con 0x7f61e00aa630 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:48.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e0144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 10 monclient(hunting): picked mon.noname-c con 0x7f61e00b5470 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.976+0000 7f61e799a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61e01461c0 0x7f61e0133c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e799a640 10 monclient(hunting): picked mon.noname-a con 0x7f61e01461c0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e799a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e799a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e01401b0 con 0x7f61e01461c0 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e799a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e799a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0140030 con 0x7f61e00aa630 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e799a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e799a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0140330 con 0x7f61e00b5470 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e799a640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e799a640 10 monclient(hunting): authenticate will time out at 1527.138466s 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e5736640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61e00aa630 0x7f61e012dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e5f37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61e01461c0 0x7f61e0133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:48.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e4f35640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e0144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e4f35640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e0144eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:45988/0 (socket says 172.21.15.49:45988) 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e4f35640 1 -- 172.21.15.49:0/2885566812 learned_addr learned my addr 172.21.15.49:0/2885566812 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e4f35640 10 monclient(hunting): get_auth_request con 0x7f61e00b5470 auth_method 0 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e4f35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e4f35640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e4f35640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e5f37640 10 monclient(hunting): get_auth_request con 0x7f61e01461c0 auth_method 0 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e5f37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e5f37640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e5f37640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e5736640 10 monclient(hunting): get_auth_request con 0x7f61e00aa630 auth_method 0 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e5736640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e5736640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.977+0000 7f61e5736640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:48.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.978+0000 7f61e4f35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.978+0000 7f61e4f35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.978+0000 7f61e4f35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.978+0000 7f61e5f37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.978+0000 7f61e5f37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.978+0000 7f61e5f37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.978+0000 7f61e5736640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.978+0000 7f61e5736640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.978+0000 7f61e5736640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e5736640 10 monclient(hunting): handle_auth_done global_id 4282 payload 824 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e5736640 1 -- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 msgr2=0x7f61e0144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e5736640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e0144eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e5736640 1 -- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61e01461c0 msgr2=0x7f61e0133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e5736640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61e01461c0 0x7f61e0133c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e5736640 10 monclient: _finish_hunting 0 2024-05-15T12:32:48.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e5736640 1 monclient: found mon.noname-b 2024-05-15T12:32:48.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e5736640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:48.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e5736640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:48.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e5736640 1 -- 172.21.15.49:0/2885566812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e0144a90 con 0x7f61e00aa630 2024-05-15T12:32:48.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.981+0000 7f61e4f35640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e0144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T12:32:48.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.983+0000 7f61e5736640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61e00aa630 0x7f61e012dc30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f61dc033000 tx=0x7f61dc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:48.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.983+0000 7f61d5ffb640 10 client.?.objecter ms_handle_connect 0x7f61e00aa630 2024-05-15T12:32:48.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.983+0000 7f61d5ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:48.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 1 -- 172.21.15.49:0/2885566812 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc032650 con 0x7f61e00aa630 2024-05-15T12:32:48.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:48.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 10 monclient: dump: 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 10 monclient: _finish_auth 0 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 1 -- 172.21.15.49:0/2885566812 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc032810 con 0x7f61e00aa630 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 1 -- 172.21.15.49:0/2885566812 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc032c00 con 0x7f61e00aa630 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d5ffb640 10 monclient: dump: 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:48.978 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61e799a640 5 monclient: authenticate success, global_id 4282 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61e799a640 10 monclient: _renew_subs 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61e799a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61e799a640 1 -- 172.21.15.49:0/2885566812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e0134b00 con 0x7f61e00aa630 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61d77fe640 4 set_mon_vals no callback set 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61e799a640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61e799a640 10 monclient: _renew_subs 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61e799a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61e799a640 1 -- 172.21.15.49:0/2885566812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e01341b0 con 0x7f61e00aa630 2024-05-15T12:32:48.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.987+0000 7f61e799a640 1 librados: init done 2024-05-15T12:32:48.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.988+0000 7f61e799a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:48.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.988+0000 7f61e799a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:48.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.988+0000 7f61e799a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:48.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:48.988+0000 7f61e799a640 1 -- 172.21.15.49:0/2885566812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e0134910 con 0x7f61e00aa630 2024-05-15T12:32:49.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.234+0000 7f61d5ffb640 1 -- 172.21.15.49:0/2885566812 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f61dc04c060 con 0x7f61e00aa630 2024-05-15T12:32:49.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.235+0000 7f61d5ffb640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f61c406cdd0 0x7f61c406f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.235+0000 7f61e4f35640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f61c406cdd0 0x7f61c406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.235+0000 7f61e4f35640 10 monclient: get_auth_request con 0x7f61c406cdd0 auth_method 0 2024-05-15T12:32:49.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.235+0000 7f61d5ffb640 10 client.4282.objecter ms_handle_connect 0x7f61c406cdd0 2024-05-15T12:32:49.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.235+0000 7f61e4f35640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f61c406cdd0 0x7f61c406f290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61e012e280 tx=0x7f61d0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.238+0000 7f61d5ffb640 1 -- 172.21.15.49:0/2885566812 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f61dc078380 con 0x7f61e00aa630 2024-05-15T12:32:49.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.238+0000 7f61d5ffb640 10 client.4282.objecter ms_dispatch 0x7f61e0000b90 osd_map(25..25 src has 1..25) v4 2024-05-15T12:32:49.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.238+0000 7f61d5ffb640 3 client.4282.objecter handle_osd_map got epochs [25,25] > 0 2024-05-15T12:32:49.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.238+0000 7f61d5ffb640 3 client.4282.objecter handle_osd_map decoding full epoch 25 2024-05-15T12:32:49.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.239+0000 7f61d5ffb640 20 client.4282.objecter dump_active .. 0 homeless 2024-05-15T12:32:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.248+0000 7f61d5ffb640 1 -- 172.21.15.49:0/2885566812 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f61e0134b00 con 0x7f61e00aa630 2024-05-15T12:32:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.248+0000 7f61d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:49.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.248+0000 7f61d5ffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:49.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.443+0000 7f61e799a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:49.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.443+0000 7f61e799a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:49.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.443+0000 7f61e799a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.443+0000 7f61e799a640 1 -- 172.21.15.49:0/2885566812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61e004f0e0 con 0x7f61e00aa630 2024-05-15T12:32:49.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.558+0000 7f61d5ffb640 1 -- 172.21.15.49:0/2885566812 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+14681 (secure 0 0 0) 0x7f61dc040070 con 0x7f61e00aa630 2024-05-15T12:32:49.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.558+0000 7f61d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T12:32:49.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.558+0000 7f61d5ffb640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:49.547 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:32:49.547 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":26,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T12:32:49.224518+0000","last_up_change":"2024-05-15T12:32:41.022082+0000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:32:30.595515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-15T12:32:45.576288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6819","nonce":3376151072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6821","nonce":3376151072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6825","nonce":3376151072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6823","nonce":3376151072}]},"public_addr":"172.21.15.49:6819/3376151072","cluster_addr":"172.21.15.49:6821/3376151072","heartbeat_back_addr":"172.21.15.49:6825/3376151072","heartbeat_front_addr":"172.21.15.49:6823/3376151072","state":["exists","up"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6811","nonce":3250791401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6813","nonce":3250791401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6817","nonce":3250791401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6815","nonce":3250791401}]},"public_addr":"172.21.15.49:6811/3250791401","cluster_addr":"172.21.15.49:6813/3250791401","heartbeat_back_addr":"172.21.15.49:6817/3250791401","heartbeat_front_addr":"172.21.15.49:6815/3250791401","state":["exists","up"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6827","nonce":2361777220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6829","nonce":2361777220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6833","nonce":2361777220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6831","nonce":2361777220}]},"public_addr":"172.21.15.49:6827/2361777220","cluster_addr":"172.21.15.49:6829/2361777220","heartbeat_back_addr":"172.21.15.49:6833/2361777220","heartbeat_front_addr":"172.21.15.49:6831/2361777220","state":["exists","up"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6803","nonce":945312262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6805","nonce":945312262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6809","nonce":945312262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6807","nonce":945312262}]},"public_addr":"172.21.15.49:6803/945312262","cluster_addr":"172.21.15.49:6805/945312262","heartbeat_back_addr":"172.21.15.49:6809/945312262","heartbeat_front_addr":"172.21.15.49:6807/945312262","state":["exists","up"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6817","nonce":3384996234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6819","nonce":3384996234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6823","nonce":3384996234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6821","nonce":3384996234}]},"public_addr":"172.21.15.88:6817/3384996234","cluster_addr":"172.21.15.88:6819/3384996234","heartbeat_back_addr":"172.21.15.88:6823/3384996234","heartbeat_front_addr":"172.21.15.88:6821/3384996234","state":["exists","up"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6809","nonce":3085185190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6811","nonce":3085185190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6815","nonce":3085185190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6813","nonce":3085185190}]},"public_addr":"172.21.15.88:6809/3085185190","cluster_addr":"172.21.15.88:6811/3085185190","heartbeat_back_addr":"172.21.15.88:6815/3085185190","heartbeat_front_addr":"172.21.15.88:6813/3085185190","state":["exists","up"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6801","nonce":3138490191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6803","nonce":3138490191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6807","nonce":3138490191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6805","nonce":3138490191}]},"public_addr":"172.21.15.88:6801/3138490191","cluster_addr":"172.21.15.88:6803/3138490191","heartbeat_back_addr":"172.21.15.88:6807/3138490191","heartbeat_front_addr":"172.21.15.88:6805/3138490191","state":["exists","up"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6825","nonce":2037618931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6827","nonce":2037618931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6831","nonce":2037618931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6829","nonce":2037618931}]},"public_addr":"172.21.15.88:6825/2037618931","cluster_addr":"172.21.15.88:6827/2037618931","heartbeat_back_addr":"172.21.15.88:6831/2037618931","heartbeat_front_addr":"172.21.15.88:6829/2037618931","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:28.806234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:29.001705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:28.402190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:27.582000+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:25.805322+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.269314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.001531+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.945037+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:32:49.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 10 librados: watch_flush enter 2024-05-15T12:32:49.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 10 librados: watch_flush exit 2024-05-15T12:32:49.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 20 client.4282.objecter shutdown clearing up homeless session... 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 10 client.4282.objecter successfully canceled tick 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 -- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f61c406cdd0 msgr2=0x7f61c406f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f61c406cdd0 0x7f61c406f290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61e012e280 tx=0x7f61d0009190 comp rx=0 tx=0).stop 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 10 monclient: shutdown 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 -- 172.21.15.49:0/2885566812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61e00aa630 msgr2=0x7f61e012dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61e00aa630 0x7f61e012dc30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f61dc033000 tx=0x7f61dc0046c0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 -- 172.21.15.49:0/2885566812 shutdown_connections 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61e01461c0 0x7f61e0133c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61e00b5470 0x7f61e0144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f61c406cdd0 0x7f61c406f290 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 --2- 172.21.15.49:0/2885566812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61e00aa630 0x7f61e012dc30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 -- 172.21.15.49:0/2885566812 >> 172.21.15.49:0/2885566812 conn(0x7f61e0066f00 msgr2=0x7f61e00a9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:49.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 -- 172.21.15.49:0/2885566812 shutdown_connections 2024-05-15T12:32:49.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 -- 172.21.15.49:0/2885566812 wait complete. 2024-05-15T12:32:49.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.560+0000 7f61e799a640 1 librados: shutdown 2024-05-15T12:32:49.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-15T12:32:49.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-15T12:32:49.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-15T12:32:49.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-15T12:32:49.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-15T12:32:49.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-15T12:32:49.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-15T12:32:49.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-15T12:32:49.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.753+0000 7ff57d531640 10 monclient: get_monmap_and_config 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.753+0000 7ff57d531640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.753+0000 7ff57d531640 10 monclient: monmap: 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:49.754039+0000 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:49.754039+0000 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:49.743 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:49.744 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:49.744 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:49.744 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fd3a0e97640 10 monclient: get_monmap_and_config 2024-05-15T12:32:49.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fd3a0e97640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fd3a0e97640 10 monclient: monmap: 2024-05-15T12:32:49.744 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.744 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.744 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:49.754567+0000 2024-05-15T12:32:49.744 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:49.754567+0000 2024-05-15T12:32:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:49.745 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7f4d9fd9a640 10 monclient: get_monmap_and_config 2024-05-15T12:32:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7f4d9fd9a640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7f4d9fd9a640 10 monclient: monmap: 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:49.755030+0000 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:49.755030+0000 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:49.746 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:49.747 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.752+0000 7fc96f4b5640 10 monclient: get_monmap_and_config 2024-05-15T12:32:49.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.752+0000 7fc96f4b5640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.752+0000 7fc96f4b5640 10 monclient: monmap: 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:49.752589+0000 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:49.752589+0000 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:49.748 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.753+0000 7fc96f4b5640 1 Processor -- start 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.753+0000 7fc96f4b5640 1 -- start start 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.753+0000 7fc96f4b5640 10 monclient: init 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc960006e80 0x7fc9600e2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 10 monclient(hunting): picked mon.noname-c con 0x7fc960006e80 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc960030d40 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc960032180 0x7fc9600e8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 10 monclient(hunting): picked mon.noname-a con 0x7fc960032180 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc960033200 con 0x7fc960030d40 2024-05-15T12:32:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc960007260 con 0x7fc960032180 2024-05-15T12:32:49.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9600312d0 con 0x7fc960006e80 2024-05-15T12:32:49.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.754+0000 7fc96f4b5640 10 monclient(hunting): authenticate will time out at 1527.915966s 2024-05-15T12:32:49.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.755+0000 7ff57d531640 1 Processor -- start 2024-05-15T12:32:49.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.755+0000 7ff57d531640 1 -- start start 2024-05-15T12:32:49.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.755+0000 7ff57d531640 10 monclient: init 2024-05-15T12:32:49.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.755+0000 7fc96e4b3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc960006e80 0x7fc9600e2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.755+0000 7fc96dcb2640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fc960030d40 0x7fc9600e2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34552/0 (socket says 172.21.15.49:34552) 2024-05-15T12:32:49.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.755+0000 7fc96dcb2640 1 -- 172.21.15.49:0/671645212 learned_addr learned my addr 172.21.15.49:0/671645212 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:49.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.755+0000 7fd3a0e97640 1 Processor -- start 2024-05-15T12:32:49.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.755+0000 7fd3a0e97640 1 -- start start 2024-05-15T12:32:49.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.755+0000 7fd3a0e97640 10 monclient: init 2024-05-15T12:32:49.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7fc32e2db640 10 monclient: get_monmap_and_config 2024-05-15T12:32:49.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7fc32e2db640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7fc32e2db640 10 monclient: monmap: 2024-05-15T12:32:49.752 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.752 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.752 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:49.756699+0000 2024-05-15T12:32:49.752 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:49.756699+0000 2024-05-15T12:32:49.752 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.752 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.753 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:49.753 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:49.753 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:49.753 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:49.753 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:49.753 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7f4d9fd9a640 1 Processor -- start 2024-05-15T12:32:49.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff57000a960 0x7ff5700349b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 10 monclient(hunting): picked mon.noname-c con 0x7ff57000a960 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5700c1120 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5700c1590 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5700327a0 con 0x7ff5700c1590 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57002fd20 con 0x7ff5700c1120 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff570031550 con 0x7ff57000a960 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7ff57d531640 10 monclient(hunting): authenticate will time out at 1527.917849s 2024-05-15T12:32:49.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7fc96e4b3640 10 monclient(hunting): get_auth_request con 0x7fc960006e80 auth_method 0 2024-05-15T12:32:49.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7fc96e4b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7fc96e4b3640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7fc96e4b3640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7f4d9fd9a640 1 -- start start 2024-05-15T12:32:49.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.756+0000 7f4d9fd9a640 10 monclient: init 2024-05-15T12:32:49.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fc96e4b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fc96e4b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd394007070 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd394007450 0x7fd3940360d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 10 monclient(hunting): picked mon.noname-a con 0x7fd394007450 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd394033a20 0x7fd3940d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 10 monclient(hunting): picked mon.noname-c con 0x7fd394033a20 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd394037140 con 0x7fd394007070 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3940c5930 con 0x7fd394007450 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd394036610 con 0x7fd394033a20 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fd3a0e97640 10 monclient(hunting): authenticate will time out at 1527.918706s 2024-05-15T12:32:49.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fc96e4b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7ff577fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff57000a960 0x7ff5700349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fc32e2db640 1 Processor -- start 2024-05-15T12:32:49.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7f4d9fd9a640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7f4d9fd9a640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-05-15T12:32:49.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7f4d9fd9a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d980b1290 0x7f4d9806fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fc32e2db640 1 -- start start 2024-05-15T12:32:49.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7fc32e2db640 10 monclient: init 2024-05-15T12:32:49.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.757+0000 7f4d9fd9a640 10 monclient(hunting): picked mon.noname-b con 0x7f4d980b1290 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4d9806ef10 0x7f4d98149f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 10 monclient(hunting): picked mon.noname-a con 0x7f4d9806ef10 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:49.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d98143d90 0x7f4d98144220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 10 monclient(hunting): picked mon.noname-c con 0x7f4d98143d90 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:49.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7ff5777fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff5700c1120 0x7ff5700c71a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:36880/0 (socket says 172.21.15.49:36880) 2024-05-15T12:32:49.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7ff5777fe640 1 -- 172.21.15.49:0/3634336011 learned_addr learned my addr 172.21.15.49:0/3634336011 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:49.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d980b1670 con 0x7f4d9806ef10 2024-05-15T12:32:49.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d9806f350 con 0x7f4d98143d90 2024-05-15T12:32:49.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9806f7d0 con 0x7f4d980b1290 2024-05-15T12:32:49.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f4d9fd9a640 10 monclient(hunting): authenticate will time out at 1527.919502s 2024-05-15T12:32:49.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7fc32e2db640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7fc32e2db640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-05-15T12:32:49.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7fc32e2db640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3280ad870 0x7fc32813d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7ff34e1aa640 10 monclient: get_monmap_and_config 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7ff34e1aa640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7ff34e1aa640 10 monclient: monmap: 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:49.759438+0000 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:49.759438+0000 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:49.763 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 10 monclient(hunting): picked mon.noname-b con 0x7fc3280ad870 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3280af260 0x7fc32813dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 10 monclient(hunting): picked mon.noname-c con 0x7fc3280af260 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3280aa6c0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3280aac70 con 0x7fc3280aa6c0 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3280adc50 con 0x7fc3280af260 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc328072fc0 con 0x7fc3280ad870 2024-05-15T12:32:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc32e2db640 10 monclient(hunting): authenticate will time out at 1527.920612s 2024-05-15T12:32:49.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f2e3cff7640 10 monclient: get_monmap_and_config 2024-05-15T12:32:49.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7f4d9db36640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d980b1290 0x7f4d9806fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96d4b1640 1 -- 172.21.15.49:0/671645212 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791238058 0 0) 0x7fc960033200 con 0x7fc960030d40 2024-05-15T12:32:49.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96d4b1640 10 monclient(hunting): _negotiate 2024-05-15T12:32:49.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96d4b1640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96d4b1640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96d4b1640 10 monclient(hunting): my global_id is 4272 2024-05-15T12:32:49.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96d4b1640 1 -- 172.21.15.49:0/671645212 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc954005730 con 0x7fc960030d40 2024-05-15T12:32:49.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.758+0000 7f2e3cff7640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7f2e3cff7640 10 monclient: monmap: 2024-05-15T12:32:49.766 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.766 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.766 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:49.759529+0000 2024-05-15T12:32:49.766 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:49.759529+0000 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7f938d1f4640 10 monclient: get_monmap_and_config 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7f938d1f4640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7f938d1f4640 10 monclient: monmap: 2024-05-15T12:32:49.767 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:49.759964+0000 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:49.759964+0000 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:49.768 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96e4b3640 10 monclient(hunting): handle_auth_done global_id 4285 payload 274 2024-05-15T12:32:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96e4b3640 1 -- 172.21.15.49:0/671645212 >> v2:172.21.15.49:3300/0 conn(0x7fc960032180 msgr2=0x7fc9600e8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96e4b3640 1 --2- 172.21.15.49:0/671645212 >> v2:172.21.15.49:3300/0 conn(0x7fc960032180 0x7fc9600e8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96e4b3640 1 -- 172.21.15.49:0/671645212 >> v1:172.21.15.49:6789/0 conn(0x7fc960030d40 legacy=0x7fc9600e2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96e4b3640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96e4b3640 1 monclient: found mon.noname-c 2024-05-15T12:32:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96e4b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96e4b3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7fc96e4b3640 1 -- 172.21.15.49:0/671645212 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600e9210 con 0x7fc960006e80 2024-05-15T12:32:49.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7ff34e1aa640 1 Processor -- start 2024-05-15T12:32:49.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.759+0000 7f4d9db36640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d980b1290 0x7f4d9806fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:45996/0 (socket says 172.21.15.49:45996) 2024-05-15T12:32:49.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f4d9db36640 1 -- 172.21.15.49:0/3431858485 learned_addr learned my addr 172.21.15.49:0/3431858485 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f4d9db36640 10 monclient(hunting): get_auth_request con 0x7f4d980b1290 auth_method 0 2024-05-15T12:32:49.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f4d9db36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f4d9db36640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f4d9db36640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7ff34e1aa640 1 -- start start 2024-05-15T12:32:49.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7ff34e1aa640 10 monclient: init 2024-05-15T12:32:49.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f2e3cff7640 1 Processor -- start 2024-05-15T12:32:49.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7ff577fff640 10 monclient(hunting): get_auth_request con 0x7ff57000a960 auth_method 0 2024-05-15T12:32:49.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7ff577fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7ff577fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7ff577fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f938d1f4640 1 Processor -- start 2024-05-15T12:32:49.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f938d1f4640 1 -- start start 2024-05-15T12:32:49.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f938d1f4640 10 monclient: init 2024-05-15T12:32:49.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f4d9db36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f4d9db36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f4d9db36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f2e3cff7640 1 -- start start 2024-05-15T12:32:49.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.760+0000 7f2e3cff7640 10 monclient: init 2024-05-15T12:32:49.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3480b5110 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff3480a9fc0 0x7ff3480b2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 10 monclient(hunting): picked mon.noname-c con 0x7ff3480a9fc0 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3480a9520 0x7ff34813f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 10 monclient(hunting): picked mon.noname-b con 0x7ff3480a9520 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3480ab220 con 0x7ff3480b5110 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3480b54f0 con 0x7ff3480a9fc0 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34813fe30 con 0x7ff3480a9520 2024-05-15T12:32:49.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff577fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff577fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff577fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-05-15T12:32:49.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7ff34e1aa640 10 monclient(hunting): authenticate will time out at 1527.922875s 2024-05-15T12:32:49.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e30034c20 0x7f2e300c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 10 monclient(hunting): picked mon.noname-c con 0x7f2e30034c20 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:49.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3002fc10 0x7f2e300c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 10 monclient(hunting): picked mon.noname-b con 0x7f2e3002fc10 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e300cf5c0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:49.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e30031d10 con 0x7f2e300cf5c0 2024-05-15T12:32:49.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300321f0 con 0x7f2e30034c20 2024-05-15T12:32:49.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300c99f0 con 0x7f2e3002fc10 2024-05-15T12:32:49.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f2e3cff7640 10 monclient(hunting): authenticate will time out at 1527.922993s 2024-05-15T12:32:49.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f938d1f4640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f938d1f4640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-05-15T12:32:49.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f938d1f4640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93880b1290 0x7f93880722e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7fd39ad9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd394033a20 0x7fd3940d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.761+0000 7f938d1f4640 10 monclient(hunting): picked mon.noname-b con 0x7f93880b1290 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.762+0000 7f938d1f4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9388141b20 0x7f938813b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.762+0000 7f938d1f4640 10 monclient(hunting): picked mon.noname-c con 0x7f9388141b20 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:49.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.762+0000 7f938d1f4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f938813bc60 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:49.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.762+0000 7f938d1f4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9388071040 con 0x7f938813bc60 2024-05-15T12:32:49.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.762+0000 7f938d1f4640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.762+0000 7f938d1f4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93880b1670 con 0x7f9388141b20 2024-05-15T12:32:49.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.762+0000 7f938d1f4640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.762+0000 7f938d1f4640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9388072820 con 0x7f93880b1290 2024-05-15T12:32:49.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.762+0000 7f938d1f4640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.762+0000 7f938d1f4640 10 monclient(hunting): authenticate will time out at 1527.923715s 2024-05-15T12:32:49.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7f4d9db36640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-05-15T12:32:49.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7f4d9db36640 1 -- 172.21.15.49:0/3431858485 >> v2:172.21.15.88:3300/0 conn(0x7f4d98143d90 msgr2=0x7f4d98144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7f4d9db36640 1 --2- 172.21.15.49:0/3431858485 >> v2:172.21.15.88:3300/0 conn(0x7f4d98143d90 0x7f4d98144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7f4d9db36640 1 -- 172.21.15.49:0/3431858485 >> v2:172.21.15.49:3300/0 conn(0x7f4d9806ef10 msgr2=0x7f4d98149f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7f4d9db36640 1 --2- 172.21.15.49:0/3431858485 >> v2:172.21.15.49:3300/0 conn(0x7f4d9806ef10 0x7f4d98149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7f4d9db36640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7f4d9db36640 1 monclient: found mon.noname-b 2024-05-15T12:32:49.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7f4d9db36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7f4d9db36640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7ff577fff640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2024-05-15T12:32:49.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7ff577fff640 1 -- 172.21.15.49:0/3634336011 >> v1:172.21.15.88:6789/0 conn(0x7ff5700c1120 legacy=0x7ff5700c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7f4d9db36640 1 -- 172.21.15.49:0/3431858485 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9806e9f0 con 0x7f4d980b1290 2024-05-15T12:32:49.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7ff577fff640 1 -- 172.21.15.49:0/3634336011 >> v1:172.21.15.49:6789/0 conn(0x7ff5700c1590 legacy=0x7ff5700c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7ff577fff640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7ff577fff640 1 monclient: found mon.noname-c 2024-05-15T12:32:49.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7ff577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7ff577fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.763+0000 7ff577fff640 1 -- 172.21.15.49:0/3634336011 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff570034ef0 con 0x7ff57000a960 2024-05-15T12:32:49.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.764+0000 7fc32d2d9640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3280ad870 0x7fc32813d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.764+0000 7fd39ad9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd394033a20 0x7fd3940d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35030/0 (socket says 172.21.15.49:35030) 2024-05-15T12:32:49.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.764+0000 7fd39ad9d640 1 -- 172.21.15.49:0/142379540 learned_addr learned my addr 172.21.15.49:0/142379540 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.764+0000 7fd39ad9d640 10 monclient(hunting): get_auth_request con 0x7fd394033a20 auth_method 0 2024-05-15T12:32:49.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.764+0000 7fd39ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.764+0000 7fd39ad9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.764+0000 7fd39ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.765+0000 7fc32d2d9640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3280ad870 0x7fc32813d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46000/0 (socket says 172.21.15.49:46000) 2024-05-15T12:32:49.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.765+0000 7fc32d2d9640 1 -- 172.21.15.49:0/2259659636 learned_addr learned my addr 172.21.15.49:0/2259659636 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.765+0000 7fc32d2d9640 10 monclient(hunting): get_auth_request con 0x7fc3280ad870 auth_method 0 2024-05-15T12:32:49.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.765+0000 7fc32d2d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.765+0000 7fc32d2d9640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.765+0000 7fc32d2d9640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.765+0000 7fd39ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.765+0000 7fd39ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.765+0000 7fd39ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.766+0000 7fc32d2d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.766+0000 7fc32d2d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.767+0000 7fc32d2d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.767+0000 7f4d9db36640 1 --2- 172.21.15.49:0/3431858485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d980b1290 0x7f4d9806fc20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4d980707a0 tx=0x7f4d9402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb7b40dea3d65f7f server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.767+0000 7ff34d9a9640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3480a9520 0x7ff34813f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.768+0000 7fd39ad9d640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2024-05-15T12:32:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.768+0000 7fd39ad9d640 1 -- 172.21.15.49:0/142379540 >> v2:172.21.15.49:3300/0 conn(0x7fd394007450 msgr2=0x7fd3940360d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.768+0000 7fd39ad9d640 1 --2- 172.21.15.49:0/142379540 >> v2:172.21.15.49:3300/0 conn(0x7fd394007450 0x7fd3940360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.768+0000 7fd39ad9d640 1 -- 172.21.15.49:0/142379540 >> v1:172.21.15.49:6789/0 conn(0x7fd394007070 legacy=0x7fd3940cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.768+0000 7fd39ad9d640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.768+0000 7fd39ad9d640 1 monclient: found mon.noname-c 2024-05-15T12:32:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.768+0000 7fd39ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.768+0000 7fd39ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.768+0000 7fd39ad9d640 1 -- 172.21.15.49:0/142379540 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3940373b0 con 0x7fd394033a20 2024-05-15T12:32:49.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.769+0000 7fc32d2d9640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-05-15T12:32:49.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.769+0000 7fc32d2d9640 1 -- 172.21.15.49:0/2259659636 >> v2:172.21.15.88:3300/0 conn(0x7fc3280af260 msgr2=0x7fc32813dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.769+0000 7fc32d2d9640 1 --2- 172.21.15.49:0/2259659636 >> v2:172.21.15.88:3300/0 conn(0x7fc3280af260 0x7fc32813dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.769+0000 7fc32d2d9640 1 -- 172.21.15.49:0/2259659636 >> v1:172.21.15.49:6789/0 conn(0x7fc3280aa6c0 legacy=0x7fc328140460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.769+0000 7fc32d2d9640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.769+0000 7fc32d2d9640 1 monclient: found mon.noname-b 2024-05-15T12:32:49.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.769+0000 7fc32d2d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.769+0000 7fc32d2d9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.769+0000 7fc32d2d9640 1 -- 172.21.15.49:0/2259659636 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc328140b70 con 0x7fc3280ad870 2024-05-15T12:32:49.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.770+0000 7f2e377fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e30034c20 0x7f2e300c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.771+0000 7f2e37fff640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f2e300cf5c0 0x7f2e300cfa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34598/0 (socket says 172.21.15.49:34598) 2024-05-15T12:32:49.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.771+0000 7f2e37fff640 1 -- 172.21.15.49:0/702393717 learned_addr learned my addr 172.21.15.49:0/702393717 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:49.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.771+0000 7ff34d9a9640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3480a9520 0x7ff34813f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46006/0 (socket says 172.21.15.49:46006) 2024-05-15T12:32:49.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.771+0000 7ff34d9a9640 1 -- 172.21.15.49:0/3792034220 learned_addr learned my addr 172.21.15.49:0/3792034220 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.771+0000 7ff34d9a9640 10 monclient(hunting): get_auth_request con 0x7ff3480a9520 auth_method 0 2024-05-15T12:32:49.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.771+0000 7ff34d9a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.771+0000 7ff34d9a9640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.771+0000 7ff34d9a9640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.771+0000 7f2e36ffd640 1 --2- 172.21.15.49:0/702393717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3002fc10 0x7f2e300c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.772+0000 7fd39ad9d640 1 --2- 172.21.15.49:0/142379540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd394033a20 0x7fd3940d25b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd39c068dc0 tx=0x7fd39c0a6680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e916cf628cbb1b33 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7f938659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9388141b20 0x7f938813b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7ff34d9a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7ff34d9a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7ff34d9a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7f2e36ffd640 10 monclient(hunting): get_auth_request con 0x7f2e3002fc10 auth_method 0 2024-05-15T12:32:49.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7f2e36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7f2e36ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7f2e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7f2e377fe640 10 monclient(hunting): get_auth_request con 0x7f2e30034c20 auth_method 0 2024-05-15T12:32:49.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7f2e377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7f2e377fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.773+0000 7f2e377fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.774+0000 7fc32d2d9640 1 --2- 172.21.15.49:0/2259659636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3280ad870 0x7fc32813d780 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc320009e50 tx=0x7fc32002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f798cf25442fe173 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.775+0000 7f2e377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.775+0000 7f2e377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.775+0000 7f2e377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.775+0000 7f938659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9388141b20 0x7f938813b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35086/0 (socket says 172.21.15.49:35086) 2024-05-15T12:32:49.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.775+0000 7f938659c640 1 -- 172.21.15.49:0/1045020306 learned_addr learned my addr 172.21.15.49:0/1045020306 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.775+0000 7f938659c640 10 monclient(hunting): get_auth_request con 0x7f9388141b20 auth_method 0 2024-05-15T12:32:49.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.775+0000 7f938659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.775+0000 7f938659c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.775+0000 7f938659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.775+0000 7fc96e4b3640 1 --2- 172.21.15.49:0/671645212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc960006e80 0x7fc9600e2890 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc960030a90 tx=0x7fc95802bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6cebc3582148042 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f2e377fe640 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2024-05-15T12:32:49.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f2e377fe640 1 -- 172.21.15.49:0/702393717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3002fc10 msgr2=0x7f2e300c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f2e377fe640 1 --2- 172.21.15.49:0/702393717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3002fc10 0x7f2e300c94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f2e377fe640 1 -- 172.21.15.49:0/702393717 >> v1:172.21.15.49:6789/0 conn(0x7f2e300cf5c0 legacy=0x7f2e300cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f2e377fe640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f2e377fe640 1 monclient: found mon.noname-c 2024-05-15T12:32:49.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f2e377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f2e377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f2e377fe640 1 -- 172.21.15.49:0/702393717 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e30032f80 con 0x7f2e30034c20 2024-05-15T12:32:49.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f938659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f938659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.777+0000 7f938659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.778+0000 7ff34d9a9640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-05-15T12:32:49.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.778+0000 7ff34d9a9640 1 -- 172.21.15.49:0/3792034220 >> v2:172.21.15.88:3300/0 conn(0x7ff3480a9fc0 msgr2=0x7ff3480b2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.778+0000 7ff34d9a9640 1 --2- 172.21.15.49:0/3792034220 >> v2:172.21.15.88:3300/0 conn(0x7ff3480a9fc0 0x7ff3480b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.778+0000 7ff34d9a9640 1 -- 172.21.15.49:0/3792034220 >> v1:172.21.15.88:6789/0 conn(0x7ff3480b5110 legacy=0x7ff3481425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.778+0000 7ff34d9a9640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.778+0000 7ff34d9a9640 1 monclient: found mon.noname-b 2024-05-15T12:32:49.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.778+0000 7ff34d9a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.778+0000 7ff34d9a9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.778+0000 7ff34d9a9640 1 -- 172.21.15.49:0/3792034220 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34813ffe0 con 0x7ff3480a9520 2024-05-15T12:32:49.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.780+0000 7f938659c640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-05-15T12:32:49.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.780+0000 7f938659c640 1 -- 172.21.15.49:0/1045020306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93880b1290 msgr2=0x7f93880722e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.780+0000 7f938659c640 1 --2- 172.21.15.49:0/1045020306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93880b1290 0x7f93880722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.780+0000 7f938659c640 1 -- 172.21.15.49:0/1045020306 >> v1:172.21.15.88:6789/0 conn(0x7f938813bc60 legacy=0x7f93880af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:49.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.780+0000 7f938659c640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.780+0000 7f938659c640 1 monclient: found mon.noname-c 2024-05-15T12:32:49.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.780+0000 7f938659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.780+0000 7f938659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.780+0000 7f938659c640 1 -- 172.21.15.49:0/1045020306 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9388071320 con 0x7f9388141b20 2024-05-15T12:32:49.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.782+0000 7ff34d9a9640 1 --2- 172.21.15.49:0/3792034220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3480a9520 0x7ff34813f8f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff340009e50 tx=0x7ff34002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a408d428ef20c2a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.786+0000 7f938659c640 1 --2- 172.21.15.49:0/1045020306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9388141b20 0x7f938813b720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f937800d1e0 tx=0x7f937802e180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fff6282ef3df287 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.788+0000 7ff33ffff640 1 -- 172.21.15.49:0/3792034220 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34002c9e0 con 0x7ff3480a9520 2024-05-15T12:32:49.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.788+0000 7ff33ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.788+0000 7ff33ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:49.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.788+0000 7ff33ffff640 10 monclient: dump: 2024-05-15T12:32:49.806 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff33ffff640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff33ffff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff33ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff33ffff640 1 -- 172.21.15.49:0/3792034220 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34002cb80 con 0x7ff3480a9520 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 5 monclient: authenticate success, global_id 4244 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff33ffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff33ffff640 1 -- 172.21.15.49:0/3792034220 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3400343f0 con 0x7ff3480a9520 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff33ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 4 set_mon_vals no callback set 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff33ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff33ffff640 10 monclient: dump: 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 10 monclient: shutdown 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 1 -- 172.21.15.49:0/3792034220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3480a9520 msgr2=0x7ff34813f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 1 --2- 172.21.15.49:0/3792034220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3480a9520 0x7ff34813f8f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff340009e50 tx=0x7ff34002be40 comp rx=0 tx=0).stop 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 1 -- 172.21.15.49:0/3792034220 shutdown_connections 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 1 --2- 172.21.15.49:0/3792034220 >> v2:172.21.15.88:3300/0 conn(0x7ff3480a9fc0 0x7ff3480b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 1 --2- 172.21.15.49:0/3792034220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3480a9520 0x7ff34813f8f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 1 -- 172.21.15.49:0/3792034220 >> 172.21.15.49:0/3792034220 conn(0x7ff34806c110 msgr2=0x7ff34806c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:49.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.789+0000 7ff34e1aa640 1 -- 172.21.15.49:0/3792034220 shutdown_connections 2024-05-15T12:32:49.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.790+0000 7ff34e1aa640 1 -- 172.21.15.49:0/3792034220 wait complete. 2024-05-15T12:32:49.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.790+0000 7f2e377fe640 1 --2- 172.21.15.49:0/702393717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30034c20 0x7f2e300c8f70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2e240097a0 tx=0x7f2e2402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab2760869fc97721 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.790+0000 7ff34e1aa640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.790+0000 7ff34e1aa640 10 monclient: monmap: 2024-05-15T12:32:49.811 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.811 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:49.811 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:49.811 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:49.811 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.811 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.811 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.790+0000 7ff34e1aa640 1 librados: starting msgr at 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7ff34e1aa640 1 librados: starting objecter 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7ff34e1aa640 1 Processor -- start 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7ff34e1aa640 1 -- start start 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7ff34e1aa640 1 librados: setting wanted keys 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7ff34e1aa640 1 librados: calling monclient init 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7ff34e1aa640 10 monclient: init 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7ff34e1aa640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7ff34e1aa640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:32:49.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 1 -- 172.21.15.49:0/3431858485 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94035020 con 0x7f4d980b1290 2024-05-15T12:32:49.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7ff34e1aa640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3480a9520 0x7ff348144450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 10 monclient: dump: 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.814 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 1 -- 172.21.15.49:0/3431858485 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d94031870 con 0x7f4d980b1290 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9fd9a640 5 monclient: authenticate success, global_id 4238 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9fd9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 1 -- 172.21.15.49:0/3431858485 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94031c40 con 0x7f4d980b1290 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9fd9a640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9fd9a640 4 set_mon_vals no callback set 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7f4d9cb34640 10 monclient: dump: 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.816 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.791+0000 7ff577fff640 1 --2- 172.21.15.49:0/3634336011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff57000a960 0x7ff5700349b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0097a0 tx=0x7ff56c02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cfa0336cfef100 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 10 monclient(hunting): picked mon.noname-a con 0x7ff3480a9520 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:49.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3480a9fc0 0x7ff348144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 10 monclient(hunting): picked mon.noname-c con 0x7ff3480a9fc0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3480b5110 0x7ff3481d7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 10 monclient(hunting): picked mon.noname-b con 0x7ff3480b5110 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:49.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34814d230 con 0x7ff3480a9520 2024-05-15T12:32:49.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34814d0b0 con 0x7ff3480b5110 2024-05-15T12:32:49.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34814d3b0 con 0x7ff3480a9fc0 2024-05-15T12:32:49.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34e1aa640 10 monclient(hunting): authenticate will time out at 1527.953827s 2024-05-15T12:32:49.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7ff34d1a8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3480a9520 0x7ff348144450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7f4d9fd9a640 10 monclient: shutdown 2024-05-15T12:32:49.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7f4d9fd9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:49.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/3431858485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d980b1290 msgr2=0x7f4d9806fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7f4d9fd9a640 1 --2- 172.21.15.49:0/3431858485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d980b1290 0x7f4d9806fc20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4d980707a0 tx=0x7f4d9402b910 comp rx=0 tx=0).stop 2024-05-15T12:32:49.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/3431858485 shutdown_connections 2024-05-15T12:32:49.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7f4d9fd9a640 1 --2- 172.21.15.49:0/3431858485 >> v2:172.21.15.88:3300/0 conn(0x7f4d98143d90 0x7f4d98144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7f4d9fd9a640 1 --2- 172.21.15.49:0/3431858485 >> v2:172.21.15.49:3300/0 conn(0x7f4d9806ef10 0x7f4d98149f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7f4d9fd9a640 1 --2- 172.21.15.49:0/3431858485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d980b1290 0x7f4d9806fc20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.792+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/3431858485 >> 172.21.15.49:0/3431858485 conn(0x7f4d980b4440 msgr2=0x7f4d980b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:49.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7ff34d1a8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3480a9520 0x7ff348144450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38938/0 (socket says 172.21.15.49:38938) 2024-05-15T12:32:49.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7ff34d1a8640 1 -- 172.21.15.49:0/2670635754 learned_addr learned my addr 172.21.15.49:0/2670635754 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7ff34d1a8640 10 monclient(hunting): get_auth_request con 0x7ff3480a9520 auth_method 0 2024-05-15T12:32:49.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7ff34d1a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7ff34d1a8640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7ff34d1a8640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/3431858485 shutdown_connections 2024-05-15T12:32:49.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/3431858485 wait complete. 2024-05-15T12:32:49.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7ff34d1a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7ff34d1a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7ff34d1a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7f4d9fd9a640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.793+0000 7f4d9fd9a640 10 monclient: monmap: 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:49.825 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 1 librados: starting msgr at 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 1 librados: starting objecter 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 1 Processor -- start 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 1 -- start start 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 1 librados: setting wanted keys 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 1 librados: calling monclient init 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 10 monclient: init 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d9806ef10 0x7f4d98149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 10 monclient(hunting): picked mon.noname-c con 0x7f4d9806ef10 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d980b1290 0x7f4d9813b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 10 monclient(hunting): picked mon.noname-b con 0x7f4d980b1290 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:49.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d98143d90 0x7f4d9813bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 10 monclient(hunting): picked mon.noname-a con 0x7f4d98143d90 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:49.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9814a620 con 0x7f4d98143d90 2024-05-15T12:32:49.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.794+0000 7f4d9fd9a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7f4d9fd9a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9814a4a0 con 0x7f4d980b1290 2024-05-15T12:32:49.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7f4d9fd9a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7f4d9fd9a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d980af820 con 0x7f4d9806ef10 2024-05-15T12:32:49.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7f4d9fd9a640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7f4d9fd9a640 10 monclient(hunting): authenticate will time out at 1527.956507s 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 1 -- 172.21.15.49:0/2259659636 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32002c9e0 con 0x7fc3280ad870 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 10 monclient: dump: 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.829 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 1 -- 172.21.15.49:0/2259659636 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32002cb80 con 0x7fc3280ad870 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc32e2db640 5 monclient: authenticate success, global_id 4241 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc32e2db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc32e2db640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc32e2db640 4 set_mon_vals no callback set 2024-05-15T12:32:49.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 1 -- 172.21.15.49:0/2259659636 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3200343f0 con 0x7fc3280ad870 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.795+0000 7fc31ffff640 10 monclient: dump: 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.831 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7fc32e2db640 10 monclient: shutdown 2024-05-15T12:32:49.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7fc32e2db640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:49.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7fc32e2db640 1 -- 172.21.15.49:0/2259659636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3280ad870 msgr2=0x7fc32813d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7fc32e2db640 1 --2- 172.21.15.49:0/2259659636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3280ad870 0x7fc32813d780 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc320009e50 tx=0x7fc32002be40 comp rx=0 tx=0).stop 2024-05-15T12:32:49.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7fc32e2db640 1 -- 172.21.15.49:0/2259659636 shutdown_connections 2024-05-15T12:32:49.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7fc32e2db640 1 --2- 172.21.15.49:0/2259659636 >> v2:172.21.15.88:3300/0 conn(0x7fc3280af260 0x7fc32813dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7fc32e2db640 1 --2- 172.21.15.49:0/2259659636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3280ad870 0x7fc32813d780 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7fc32e2db640 1 -- 172.21.15.49:0/2259659636 >> 172.21.15.49:0/2259659636 conn(0x7fc3280b40d0 msgr2=0x7fc3280b44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:49.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7fc32e2db640 1 -- 172.21.15.49:0/2259659636 shutdown_connections 2024-05-15T12:32:49.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7fc32e2db640 1 -- 172.21.15.49:0/2259659636 wait complete. 2024-05-15T12:32:49.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7ff34d1a8640 10 monclient(hunting): handle_auth_done global_id 4275 payload 824 2024-05-15T12:32:49.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7ff34d1a8640 1 -- 172.21.15.49:0/2670635754 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3480a9fc0 msgr2=0x7ff348144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7ff34d1a8640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3480a9fc0 0x7ff348144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7ff34d1a8640 1 -- 172.21.15.49:0/2670635754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3480b5110 msgr2=0x7ff3481d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7ff34d1a8640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3480b5110 0x7ff3481d7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7ff34d1a8640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7ff34d1a8640 1 monclient: found mon.noname-a 2024-05-15T12:32:49.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7ff34d1a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7ff34d1a8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:49.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.796+0000 7ff34d1a8640 1 -- 172.21.15.49:0/2670635754 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff348143eb0 con 0x7ff3480a9520 2024-05-15T12:32:49.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 1 -- 172.21.15.49:0/142379540 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c0b2070 con 0x7fd394033a20 2024-05-15T12:32:49.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:49.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 10 monclient: dump: 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 1 -- 172.21.15.49:0/142379540 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c0ab7d0 con 0x7fd394033a20 2024-05-15T12:32:49.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 1 -- 172.21.15.49:0/142379540 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c0abc00 con 0x7fd394033a20 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd39959a640 10 monclient: dump: 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.836 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.837 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.837 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.837 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd3a0e97640 5 monclient: authenticate success, global_id 4291 2024-05-15T12:32:49.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd3a0e97640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:49.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fd3a0e97640 4 set_mon_vals no callback set 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fc32e2db640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fc32e2db640 10 monclient: monmap: 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fc32e2db640 1 librados: starting msgr at 2024-05-15T12:32:49.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.797+0000 7fc32e2db640 1 librados: starting objecter 2024-05-15T12:32:49.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fc32e2db640 1 Processor -- start 2024-05-15T12:32:49.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fd3a0e97640 10 monclient: shutdown 2024-05-15T12:32:49.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fd3a0e97640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:49.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fd3a0e97640 1 -- 172.21.15.49:0/142379540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd394033a20 msgr2=0x7fd3940d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fd3a0e97640 1 --2- 172.21.15.49:0/142379540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd394033a20 0x7fd3940d25b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd39c068dc0 tx=0x7fd39c0a6680 comp rx=0 tx=0).stop 2024-05-15T12:32:49.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 1 -- 172.21.15.49:0/1045020306 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9378034070 con 0x7f9388141b20 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 10 monclient: dump: 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 1 -- 172.21.15.49:0/1045020306 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9378038400 con 0x7f9388141b20 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 1 -- 172.21.15.49:0/1045020306 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93780387f0 con 0x7f9388141b20 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f9385d9b640 10 monclient: dump: 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.842 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.842 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f938d1f4640 5 monclient: authenticate success, global_id 4297 2024-05-15T12:32:49.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f938d1f4640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:49.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f938d1f4640 4 set_mon_vals no callback set 2024-05-15T12:32:49.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fc32e2db640 1 -- start start 2024-05-15T12:32:49.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fc32e2db640 1 librados: setting wanted keys 2024-05-15T12:32:49.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fc32e2db640 1 librados: calling monclient init 2024-05-15T12:32:49.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fc32e2db640 10 monclient: init 2024-05-15T12:32:49.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fd3a0e97640 1 -- 172.21.15.49:0/142379540 shutdown_connections 2024-05-15T12:32:49.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fd3a0e97640 1 --2- 172.21.15.49:0/142379540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd394033a20 0x7fd3940d25b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fd3a0e97640 1 --2- 172.21.15.49:0/142379540 >> v2:172.21.15.49:3300/0 conn(0x7fd394007450 0x7fd3940360d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fd3a0e97640 1 -- 172.21.15.49:0/142379540 >> 172.21.15.49:0/142379540 conn(0x7fd394006250 msgr2=0x7fd394006660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:49.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fd3a0e97640 1 -- 172.21.15.49:0/142379540 shutdown_connections 2024-05-15T12:32:49.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7fd3a0e97640 1 -- 172.21.15.49:0/142379540 wait complete. 2024-05-15T12:32:49.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.798+0000 7f4d9db36640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d9806ef10 0x7f4d98149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7f938d1f4640 10 monclient: shutdown 2024-05-15T12:32:49.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7f938d1f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:49.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7f938d1f4640 1 -- 172.21.15.49:0/1045020306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9388141b20 msgr2=0x7f938813b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7f938d1f4640 1 --2- 172.21.15.49:0/1045020306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9388141b20 0x7f938813b720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f937800d1e0 tx=0x7f937802e180 comp rx=0 tx=0).stop 2024-05-15T12:32:49.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7f938d1f4640 1 -- 172.21.15.49:0/1045020306 shutdown_connections 2024-05-15T12:32:49.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7f938d1f4640 1 --2- 172.21.15.49:0/1045020306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9388141b20 0x7f938813b720 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7f938d1f4640 1 --2- 172.21.15.49:0/1045020306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93880b1290 0x7f93880722e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7f938d1f4640 1 -- 172.21.15.49:0/1045020306 >> 172.21.15.49:0/1045020306 conn(0x7f93880b4440 msgr2=0x7f93880b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:49.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:32:49.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3280aa6c0 0x7fc32813ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7f938d1f4640 1 -- 172.21.15.49:0/1045020306 shutdown_connections 2024-05-15T12:32:49.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 10 monclient(hunting): picked mon.noname-c con 0x7fc3280aa6c0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3280ad870 0x7fc3281d6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 10 monclient(hunting): picked mon.noname-b con 0x7fc3280ad870 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:49.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3280af260 0x7fc3281d6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7f938d1f4640 1 -- 172.21.15.49:0/1045020306 wait complete. 2024-05-15T12:32:49.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 10 monclient(hunting): picked mon.noname-a con 0x7fc3280af260 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:49.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32814ac90 con 0x7fc3280af260 2024-05-15T12:32:49.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3281410c0 con 0x7fc3280ad870 2024-05-15T12:32:49.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32814ae10 con 0x7fc3280aa6c0 2024-05-15T12:32:49.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fc32e2db640 10 monclient(hunting): authenticate will time out at 1527.960901s 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fd3a0e97640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.799+0000 7fd3a0e97640 10 monclient: monmap: 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7fd3a0e97640 1 librados: starting msgr at 2024-05-15T12:32:49.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7f4d9db36640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d9806ef10 0x7f4d98149cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46028/0 (socket says 172.21.15.49:46028) 2024-05-15T12:32:49.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7f4d9db36640 1 -- 172.21.15.49:0/566425273 learned_addr learned my addr 172.21.15.49:0/566425273 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7f4d9db36640 10 monclient(hunting): get_auth_request con 0x7f4d9806ef10 auth_method 0 2024-05-15T12:32:49.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7f4d9db36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7f4d9db36640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7f4d9db36640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7fd3a0e97640 1 librados: starting objecter 2024-05-15T12:32:49.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7f938d1f4640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7fd3a0e97640 1 Processor -- start 2024-05-15T12:32:49.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7fd3a0e97640 1 -- start start 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7f938d1f4640 10 monclient: monmap: 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:49.852 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7fd3a0e97640 1 librados: setting wanted keys 2024-05-15T12:32:49.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7fd3a0e97640 1 librados: calling monclient init 2024-05-15T12:32:49.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.800+0000 7fd3a0e97640 10 monclient: init 2024-05-15T12:32:49.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.801+0000 7f938d1f4640 1 librados: starting msgr at 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.801+0000 7f938d1f4640 1 librados: starting objecter 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.801+0000 7f938d1f4640 1 Processor -- start 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7fd3a0e97640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f938d1f4640 1 -- start start 2024-05-15T12:32:49.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f938d1f4640 1 librados: setting wanted keys 2024-05-15T12:32:49.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f938d1f4640 1 librados: calling monclient init 2024-05-15T12:32:49.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f938d1f4640 10 monclient: init 2024-05-15T12:32:49.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7fd3a0e97640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:32:49.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7fd3a0e97640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd394007070 0x7fd3941609a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f4d9db36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f4d9db36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f4d9db36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7ff34d1a8640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3480a9520 0x7ff348144450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff34400b3e0 tx=0x7ff34400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7ff33dffb640 10 client.?.objecter ms_handle_connect 0x7ff3480a9520 2024-05-15T12:32:49.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7ff33dffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:49.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7fd3a0e97640 10 monclient(hunting): picked mon.noname-c con 0x7fd394007070 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7fd3a0e97640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd394007450 0x7fd394161120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 10 monclient(hunting): picked mon.noname-a con 0x7fd394007450 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:49.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd394033a20 0x7fd3940c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:32:49.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93880b1290 0x7f938813d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 10 monclient(hunting): picked mon.noname-b con 0x7fd394033a20 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 1 -- 172.21.15.49:0/702393717 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24035020 con 0x7f2e30034c20 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 10 monclient: dump: 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.859 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e3cff7640 5 monclient: authenticate success, global_id 4294 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e3cff7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 1 -- 172.21.15.49:0/702393717 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e24031830 con 0x7f2e30034c20 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e3cff7640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:49.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 10 monclient(hunting): picked mon.noname-c con 0x7f93880b1290 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e3cff7640 4 set_mon_vals no callback set 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 1 -- 172.21.15.49:0/702393717 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24031c00 con 0x7f2e30034c20 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.802+0000 7f2e367fc640 10 monclient: dump: 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.862 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.862 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.862 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.862 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f2e3cff7640 10 monclient: shutdown 2024-05-15T12:32:49.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f2e3cff7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:49.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f2e3cff7640 1 -- 172.21.15.49:0/702393717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30034c20 msgr2=0x7f2e300c8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f2e3cff7640 1 --2- 172.21.15.49:0/702393717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30034c20 0x7f2e300c8f70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2e240097a0 tx=0x7f2e2402b8d0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f938813bc60 0x7f938813d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96d4b1640 1 -- 172.21.15.49:0/671645212 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95802d8e0 con 0x7fc960006e80 2024-05-15T12:32:49.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96d4b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96d4b1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:49.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96d4b1640 10 monclient: dump: 2024-05-15T12:32:49.863 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.863 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96d4b1640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96d4b1640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96d4b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96f4b5640 5 monclient: authenticate success, global_id 4285 2024-05-15T12:32:49.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96f4b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:49.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96d4b1640 1 -- 172.21.15.49:0/671645212 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95800c040 con 0x7fc960006e80 2024-05-15T12:32:49.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96d4b1640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96f4b5640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:49.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fc96f4b5640 4 set_mon_vals no callback set 2024-05-15T12:32:49.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 10 monclient(hunting): picked mon.noname-a con 0x7f938813bc60 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:49.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd394035bb0 con 0x7fd394007450 2024-05-15T12:32:49.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9388141b20 0x7f93881d70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd394035950 con 0x7fd394033a20 2024-05-15T12:32:49.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd394030c20 con 0x7fd394007070 2024-05-15T12:32:49.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 10 monclient(hunting): picked mon.noname-b con 0x7f9388141b20 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:49.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7fd3a0e97640 10 monclient(hunting): authenticate will time out at 1527.965156s 2024-05-15T12:32:49.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9388073030 con 0x7f938813bc60 2024-05-15T12:32:49.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9388071dc0 con 0x7f9388141b20 2024-05-15T12:32:49.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9388071a20 con 0x7f93880b1290 2024-05-15T12:32:49.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.803+0000 7f938d1f4640 10 monclient(hunting): authenticate will time out at 1527.965273s 2024-05-15T12:32:49.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7f2e3cff7640 1 -- 172.21.15.49:0/702393717 shutdown_connections 2024-05-15T12:32:49.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7f2e3cff7640 1 --2- 172.21.15.49:0/702393717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3002fc10 0x7f2e300c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc32dada640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3280af260 0x7fc3281d6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7f2e3cff7640 1 --2- 172.21.15.49:0/702393717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30034c20 0x7f2e300c8f70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7f2e3cff7640 1 -- 172.21.15.49:0/702393717 >> 172.21.15.49:0/702393717 conn(0x7f2e300060c0 msgr2=0x7f2e300064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96d4b1640 1 -- 172.21.15.49:0/671645212 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95802dce0 con 0x7fc960006e80 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96d4b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96d4b1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96d4b1640 10 monclient: dump: 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.870 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.871 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.871 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.871 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.871 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7f2e3cff7640 1 -- 172.21.15.49:0/702393717 shutdown_connections 2024-05-15T12:32:49.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7f2e3cff7640 1 -- 172.21.15.49:0/702393717 wait complete. 2024-05-15T12:32:49.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96f4b5640 10 monclient: shutdown 2024-05-15T12:32:49.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96f4b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:49.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96f4b5640 1 -- 172.21.15.49:0/671645212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc960006e80 msgr2=0x7fc9600e2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96f4b5640 1 --2- 172.21.15.49:0/671645212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc960006e80 0x7fc9600e2890 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc960030a90 tx=0x7fc95802bb20 comp rx=0 tx=0).stop 2024-05-15T12:32:49.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96f4b5640 1 -- 172.21.15.49:0/671645212 shutdown_connections 2024-05-15T12:32:49.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96f4b5640 1 --2- 172.21.15.49:0/671645212 >> v2:172.21.15.49:3300/0 conn(0x7fc960032180 0x7fc9600e8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96f4b5640 1 --2- 172.21.15.49:0/671645212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc960006e80 0x7fc9600e2890 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96f4b5640 1 -- 172.21.15.49:0/671645212 >> 172.21.15.49:0/671645212 conn(0x7fc960005ee0 msgr2=0x7fc9600062f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:49.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96f4b5640 1 -- 172.21.15.49:0/671645212 shutdown_connections 2024-05-15T12:32:49.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7fc96f4b5640 1 -- 172.21.15.49:0/671645212 wait complete. 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7f2e3cff7640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.804+0000 7f2e3cff7640 10 monclient: monmap: 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:49.873 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7fc96f4b5640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7fc96f4b5640 10 monclient: monmap: 2024-05-15T12:32:49.874 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.874 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:49.874 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:49.874 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:49.874 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.874 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.874 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:49.874 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:49.875 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:49.875 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7f2e3cff7640 1 librados: starting msgr at 2024-05-15T12:32:49.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7f2e3cff7640 1 librados: starting objecter 2024-05-15T12:32:49.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 1 -- 172.21.15.49:0/3634336011 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c038070 con 0x7ff57000a960 2024-05-15T12:32:49.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:49.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 10 monclient: dump: 2024-05-15T12:32:49.876 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.876 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.876 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.876 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.876 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 1 -- 172.21.15.49:0/3634336011 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c031700 con 0x7ff57000a960 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 1 -- 172.21.15.49:0/3634336011 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c031ad0 con 0x7ff57000a960 2024-05-15T12:32:49.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff576ffd640 10 monclient: dump: 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff57d531640 5 monclient: authenticate success, global_id 4288 2024-05-15T12:32:49.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff57d531640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:49.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7f2e3cff7640 1 Processor -- start 2024-05-15T12:32:49.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7ff57d531640 4 set_mon_vals no callback set 2024-05-15T12:32:49.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7fc96f4b5640 1 librados: starting msgr at 2024-05-15T12:32:49.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7fc96f4b5640 1 librados: starting objecter 2024-05-15T12:32:49.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7f2e3cff7640 1 -- start start 2024-05-15T12:32:49.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7f2e3cff7640 1 librados: setting wanted keys 2024-05-15T12:32:49.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7f2e3cff7640 1 librados: calling monclient init 2024-05-15T12:32:49.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7f2e3cff7640 10 monclient: init 2024-05-15T12:32:49.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7fc96f4b5640 1 Processor -- start 2024-05-15T12:32:49.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7fc96f4b5640 1 -- start start 2024-05-15T12:32:49.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7fc96f4b5640 1 librados: setting wanted keys 2024-05-15T12:32:49.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7fc96f4b5640 1 librados: calling monclient init 2024-05-15T12:32:49.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.805+0000 7fc96f4b5640 10 monclient: init 2024-05-15T12:32:49.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7ff57d531640 10 monclient: shutdown 2024-05-15T12:32:49.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7ff57d531640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:49.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7ff57d531640 1 -- 172.21.15.49:0/3634336011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff57000a960 msgr2=0x7ff5700349b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7ff57d531640 1 --2- 172.21.15.49:0/3634336011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff57000a960 0x7ff5700349b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0097a0 tx=0x7ff56c02b7a0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:32:49.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e3002fc10 0x7f2e300cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7ff57d531640 1 -- 172.21.15.49:0/3634336011 shutdown_connections 2024-05-15T12:32:49.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7ff57d531640 1 --2- 172.21.15.49:0/3634336011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff57000a960 0x7ff5700349b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7ff57d531640 1 -- 172.21.15.49:0/3634336011 >> 172.21.15.49:0/3634336011 conn(0x7ff5700060c0 msgr2=0x7ff5700064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:49.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 10 monclient(hunting): picked mon.noname-a con 0x7f2e3002fc10 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:49.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30034c20 0x7f2e300cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 10 monclient(hunting): picked mon.noname-b con 0x7f2e30034c20 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e300cf5c0 0x7f2e300cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 10 monclient(hunting): picked mon.noname-c con 0x7f2e300cf5c0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30035180 con 0x7f2e3002fc10 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30035000 con 0x7f2e30034c20 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300d6950 con 0x7f2e300cf5c0 2024-05-15T12:32:49.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7f2e3cff7640 10 monclient(hunting): authenticate will time out at 1527.967768s 2024-05-15T12:32:49.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc960006e80 0x7fc9600db950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 10 monclient(hunting): picked mon.noname-c con 0x7fc960006e80 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc960030d40 0x7fc9600d6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 10 monclient(hunting): picked mon.noname-b con 0x7fc960030d40 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:49.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc960032180 0x7fc96017f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 10 monclient(hunting): picked mon.noname-a con 0x7fc960032180 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:49.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600e9870 con 0x7fc960032180 2024-05-15T12:32:49.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600e96f0 con 0x7fc960030d40 2024-05-15T12:32:49.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96017ecb0 con 0x7fc960006e80 2024-05-15T12:32:49.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7fc96f4b5640 10 monclient(hunting): authenticate will time out at 1527.967857s 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7ff57d531640 1 -- 172.21.15.49:0/3634336011 shutdown_connections 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.806+0000 7ff57d531640 1 -- 172.21.15.49:0/3634336011 wait complete. 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7ff57d531640 10 monclient: build_initial_monmap 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7ff57d531640 10 monclient: monmap: 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.887 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:49.888 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:49.888 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:49.888 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9d335640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d980b1290 0x7f4d9813b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7fc32dada640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3280af260 0x7fc3281d6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38948/0 (socket says 172.21.15.49:38948) 2024-05-15T12:32:49.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7fc32dada640 1 -- 172.21.15.49:0/94651508 learned_addr learned my addr 172.21.15.49:0/94651508 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7fc32dada640 10 monclient(hunting): get_auth_request con 0x7fc3280af260 auth_method 0 2024-05-15T12:32:49.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7fc32dada640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7fc32dada640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7fc32dada640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9e337640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d98143d90 0x7f4d9813bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9d335640 10 monclient(hunting): get_auth_request con 0x7f4d980b1290 auth_method 0 2024-05-15T12:32:49.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9d335640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9d335640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9d335640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7ff57d531640 1 librados: starting msgr at 2024-05-15T12:32:49.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7ff57d531640 1 librados: starting objecter 2024-05-15T12:32:49.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7fd399d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd394007450 0x7fd394161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7fc32cad8640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3280ad870 0x7fc3281d6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9db36640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-05-15T12:32:49.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7ff57d531640 1 Processor -- start 2024-05-15T12:32:49.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9db36640 1 -- 172.21.15.49:0/566425273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d980b1290 msgr2=0x7f4d9813b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9db36640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d980b1290 0x7f4d9813b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9db36640 1 -- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d98143d90 msgr2=0x7f4d9813bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9db36640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d98143d90 0x7f4d9813bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9db36640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.807+0000 7f4d9db36640 1 monclient: found mon.noname-c 2024-05-15T12:32:49.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.808+0000 7f4d9db36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.808+0000 7f4d9db36640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.808+0000 7f4d9db36640 1 -- 172.21.15.49:0/566425273 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d980aa410 con 0x7f4d9806ef10 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.808+0000 7ff57d531640 1 -- start start 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.808+0000 7ff57d531640 1 librados: setting wanted keys 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.808+0000 7ff57d531640 1 librados: calling monclient init 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.808+0000 7ff57d531640 10 monclient: init 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57000a960 0x7ff5700c0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 10 monclient(hunting): picked mon.noname-c con 0x7ff57000a960 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5700c1120 0x7ff57002b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 10 monclient(hunting): picked mon.noname-a con 0x7ff5700c1120 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5700c1590 0x7ff5700c3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 10 monclient(hunting): picked mon.noname-b con 0x7ff5700c1590 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:49.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff570031330 con 0x7ff5700c1120 2024-05-15T12:32:49.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5700311b0 con 0x7ff5700c1590 2024-05-15T12:32:49.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:49.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff570031f70 con 0x7ff57000a960 2024-05-15T12:32:49.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:49.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7fc32dada640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7fc32dada640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7ff57d531640 10 monclient(hunting): authenticate will time out at 1527.971053s 2024-05-15T12:32:49.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7fc32dada640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7fc32cad8640 10 monclient(hunting): get_auth_request con 0x7fc3280ad870 auth_method 0 2024-05-15T12:32:49.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7fc32cad8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7fc32cad8640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7fc32cad8640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.809+0000 7f938759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9388141b20 0x7f93881d70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7fd39ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd394033a20 0x7fd3940c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7f938659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f938813bc60 0x7f938813d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7f4d8e7fc640 10 client.?.objecter ms_handle_connect 0x7f4d9806ef10 2024-05-15T12:32:49.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7f4d8e7fc640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:49.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7f4d9db36640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d9806ef10 0x7f4d98149cb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4d94000c00 tx=0x7f4d940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7f938759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9388141b20 0x7f93881d70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35140/0 (socket says 172.21.15.49:35140) 2024-05-15T12:32:49.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7f938759e640 1 -- 172.21.15.49:0/862655462 learned_addr learned my addr 172.21.15.49:0/862655462 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7f938759e640 10 monclient(hunting): get_auth_request con 0x7f9388141b20 auth_method 0 2024-05-15T12:32:49.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7f938759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7f938759e640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.810+0000 7f938759e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7fd399d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd394007450 0x7fd394161120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38962/0 (socket says 172.21.15.49:38962) 2024-05-15T12:32:49.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7fd399d9b640 1 -- 172.21.15.49:0/3521073038 learned_addr learned my addr 172.21.15.49:0/3521073038 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7fd399d9b640 10 monclient(hunting): get_auth_request con 0x7fd394007450 auth_method 0 2024-05-15T12:32:49.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7fd399d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7fd399d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7fd399d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7f938759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7f938759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7f938759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7f938659c640 10 monclient(hunting): get_auth_request con 0x7f938813bc60 auth_method 0 2024-05-15T12:32:49.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7f938659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7f938659c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.811+0000 7f938659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.812+0000 7fd39ad9d640 10 monclient(hunting): get_auth_request con 0x7fd394033a20 auth_method 0 2024-05-15T12:32:49.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.812+0000 7fd39ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.812+0000 7fd39ad9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.812+0000 7fd39ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.812+0000 7fd39ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.812+0000 7fd39ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.812+0000 7fd39ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.814+0000 7fc32dada640 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-05-15T12:32:49.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.814+0000 7fc32dada640 1 -- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3280aa6c0 msgr2=0x7fc32813ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.814+0000 7fc32dada640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3280aa6c0 0x7fc32813ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.814+0000 7fc32dada640 1 -- 172.21.15.49:0/94651508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3280ad870 msgr2=0x7fc3281d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.814+0000 7fc32dada640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3280ad870 0x7fc3281d6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.814+0000 7fc32dada640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.814+0000 7fc32dada640 1 monclient: found mon.noname-a 2024-05-15T12:32:49.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.814+0000 7fc32dada640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.814+0000 7fc32dada640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:49.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.814+0000 7fc32dada640 1 -- 172.21.15.49:0/94651508 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc328138210 con 0x7fc3280af260 2024-05-15T12:32:49.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7f938759e640 10 monclient(hunting): handle_auth_done global_id 4303 payload 824 2024-05-15T12:32:49.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7f938759e640 1 -- 172.21.15.49:0/862655462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93880b1290 msgr2=0x7f938813d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7f938759e640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93880b1290 0x7f938813d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7f938759e640 1 -- 172.21.15.49:0/862655462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f938813bc60 msgr2=0x7f938813d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7f938759e640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f938813bc60 0x7f938813d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7f938759e640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7f938759e640 1 monclient: found mon.noname-b 2024-05-15T12:32:49.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7f938759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7f938759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7f938759e640 1 -- 172.21.15.49:0/862655462 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938813f8e0 con 0x7f9388141b20 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7fd39a59c640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd394007070 0x7fd3941609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7fd399d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7fd399d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7fd399d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7fd39a59c640 10 monclient(hunting): get_auth_request con 0x7fd394007070 auth_method 0 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7fd39a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7fd39a59c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.815+0000 7fd39a59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.816+0000 7fd39a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.816+0000 7fd39a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.816+0000 7fd39a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.819+0000 7fd39a59c640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-05-15T12:32:49.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.819+0000 7fd39a59c640 1 -- 172.21.15.49:0/3521073038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd394033a20 msgr2=0x7fd3940c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.819+0000 7fd39a59c640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd394033a20 0x7fd3940c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.819+0000 7fd39a59c640 1 -- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd394007450 msgr2=0x7fd394161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.819+0000 7fd39a59c640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd394007450 0x7fd394161120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.819+0000 7fd39a59c640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.819+0000 7fd39a59c640 1 monclient: found mon.noname-c 2024-05-15T12:32:49.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.819+0000 7fd39a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.819+0000 7fd39a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.819+0000 7fd39a59c640 1 -- 172.21.15.49:0/3521073038 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd394030130 con 0x7fd394007070 2024-05-15T12:32:49.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.821+0000 7f938759e640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9388141b20 0x7f93881d70d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f937000c390 tx=0x7f937000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.821+0000 7f937f7fe640 10 client.?.objecter ms_handle_connect 0x7f9388141b20 2024-05-15T12:32:49.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.821+0000 7f937f7fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:49.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.821+0000 7fc32dada640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3280af260 0x7fc3281d6790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc31800c410 tx=0x7fc31800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.821+0000 7fc31dffb640 10 client.?.objecter ms_handle_connect 0x7fc3280af260 2024-05-15T12:32:49.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.822+0000 7fc31dffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:49.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.823+0000 7fd39a59c640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd394007070 0x7fd3941609a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c410 tx=0x7fd39000c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.823+0000 7fd38affd640 10 client.?.objecter ms_handle_connect 0x7fd394007070 2024-05-15T12:32:49.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.823+0000 7fd38affd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:49.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 1 -- 172.21.15.49:0/2670635754 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344015020 con 0x7ff3480a9520 2024-05-15T12:32:49.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:49.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 10 monclient: dump: 2024-05-15T12:32:49.910 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.910 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.910 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.910 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 1 -- 172.21.15.49:0/2670635754 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff344011850 con 0x7ff3480a9520 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 1 -- 172.21.15.49:0/2670635754 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344011c40 con 0x7ff3480a9520 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33dffb640 10 monclient: dump: 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff34e1aa640 5 monclient: authenticate success, global_id 4275 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff33f7fe640 4 set_mon_vals no callback set 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff34e1aa640 10 monclient: _renew_subs 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff34e1aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:49.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3481453d0 con 0x7ff3480a9520 2024-05-15T12:32:49.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff34e1aa640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff34e1aa640 10 monclient: _renew_subs 2024-05-15T12:32:49.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff34e1aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:49.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3481d8360 con 0x7ff3480a9520 2024-05-15T12:32:49.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7ff34e1aa640 1 librados: init done 2024-05-15T12:32:49.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.825+0000 7f2e37fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e300cf5c0 0x7f2e300cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7fc96e4b3640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc960006e80 0x7fc9600db950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 1 -- 172.21.15.49:0/566425273 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9403ad60 con 0x7f4d9806ef10 2024-05-15T12:32:49.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:49.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 10 monclient: dump: 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 1 -- 172.21.15.49:0/566425273 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d940443f0 con 0x7f4d9806ef10 2024-05-15T12:32:49.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 1 -- 172.21.15.49:0/566425273 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d940447e0 con 0x7f4d9806ef10 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8e7fc640 10 monclient: dump: 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.916 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.917 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d9fd9a640 5 monclient: authenticate success, global_id 4250 2024-05-15T12:32:49.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d9fd9a640 10 monclient: _renew_subs 2024-05-15T12:32:49.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d8ffff640 4 set_mon_vals no callback set 2024-05-15T12:32:49.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7ff34e1aa640 10 client.4275.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:49.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7ff34e1aa640 15 client.4275.objecter _session_command_op_assign -1 1 2024-05-15T12:32:49.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7ff34e1aa640 20 client.4275.objecter _get_session osd=-1 returning homeless 2024-05-15T12:32:49.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7ff34e1aa640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7ff34e1aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:49.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff348146630 con 0x7ff3480a9520 2024-05-15T12:32:49.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d9fd9a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d980aa680 con 0x7f4d9806ef10 2024-05-15T12:32:49.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7fc96dcb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc960030d40 0x7fc9600d6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d9fd9a640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d9fd9a640 10 monclient: _renew_subs 2024-05-15T12:32:49.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d9fd9a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d981d36a0 con 0x7f4d9806ef10 2024-05-15T12:32:49.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.826+0000 7f4d9fd9a640 1 librados: init done 2024-05-15T12:32:49.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7fc96dcb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc960030d40 0x7fc9600d6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35162/0 (socket says 172.21.15.49:35162) 2024-05-15T12:32:49.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7fc96dcb2640 1 -- 172.21.15.49:0/650259672 learned_addr learned my addr 172.21.15.49:0/650259672 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7fc96dcb2640 10 monclient(hunting): get_auth_request con 0x7fc960030d40 auth_method 0 2024-05-15T12:32:49.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7fc96dcb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7fc96dcb2640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7fc96dcb2640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7f4d9fd9a640 10 client.4250.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:49.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7f4d9fd9a640 15 client.4250.objecter _session_command_op_assign -1 1 2024-05-15T12:32:49.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7f4d9fd9a640 20 client.4250.objecter _get_session osd=-1 returning homeless 2024-05-15T12:32:49.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7f4d9fd9a640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7f4d9fd9a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.827+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4d981d3870 con 0x7f4d9806ef10 2024-05-15T12:32:49.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7ff577fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57000a960 0x7ff5700c0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7fc96e4b3640 10 monclient(hunting): get_auth_request con 0x7fc960006e80 auth_method 0 2024-05-15T12:32:49.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7fc96e4b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7fc96e4b3640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7fc96e4b3640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7f2e36ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30034c20 0x7f2e300cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7f2e37fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e300cf5c0 0x7f2e300cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46060/0 (socket says 172.21.15.49:46060) 2024-05-15T12:32:49.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7f2e37fff640 1 -- 172.21.15.49:0/2760587583 learned_addr learned my addr 172.21.15.49:0/2760587583 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7f2e37fff640 10 monclient(hunting): get_auth_request con 0x7f2e300cf5c0 auth_method 0 2024-05-15T12:32:49.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7f2e37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7f2e37fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.828+0000 7f2e37fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7fc96dcb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7fc96dcb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7fc96dcb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7fc96e4b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7fc96e4b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7fc96e4b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7f2e36ffd640 10 monclient(hunting): get_auth_request con 0x7f2e30034c20 auth_method 0 2024-05-15T12:32:49.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7f2e36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7f2e36ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7f2e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7f2e37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7f2e37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7f2e37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.830+0000 7ff577fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57000a960 0x7ff5700c0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46082/0 (socket says 172.21.15.49:46082) 2024-05-15T12:32:49.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7ff577fff640 1 -- 172.21.15.49:0/2673634926 learned_addr learned my addr 172.21.15.49:0/2673634926 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:49.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7ff577fff640 10 monclient(hunting): get_auth_request con 0x7ff57000a960 auth_method 0 2024-05-15T12:32:49.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7ff577fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7ff577fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7ff577fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 1 -- 172.21.15.49:0/94651508 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318011790 con 0x7fc3280af260 2024-05-15T12:32:49.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:49.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 10 monclient: dump: 2024-05-15T12:32:49.927 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 1 -- 172.21.15.49:0/94651508 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318017070 con 0x7fc3280af260 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc32e2db640 5 monclient: authenticate success, global_id 4278 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc32e2db640 10 monclient: _renew_subs 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc32e2db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc32e2db640 1 -- 172.21.15.49:0/94651508 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3281384f0 con 0x7fc3280af260 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 1 -- 172.21.15.49:0/94651508 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318011bb0 con 0x7fc3280af260 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31dffb640 10 monclient: dump: 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.929 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.930 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.930 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.930 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.930 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.930 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc96ecb4640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc960032180 0x7fc96017f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.831+0000 7fc31f7fe640 4 set_mon_vals no callback set 2024-05-15T12:32:49.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.832+0000 7fc32e2db640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.832+0000 7fc32e2db640 10 monclient: _renew_subs 2024-05-15T12:32:49.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.832+0000 7fc32e2db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:49.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.832+0000 7fc32e2db640 1 -- 172.21.15.49:0/94651508 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32813f0f0 con 0x7fc3280af260 2024-05-15T12:32:49.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.832+0000 7fc32e2db640 1 librados: init done 2024-05-15T12:32:49.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.832+0000 7fc96ecb4640 10 monclient(hunting): get_auth_request con 0x7fc960032180 auth_method 0 2024-05-15T12:32:49.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.832+0000 7fc96ecb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.832+0000 7fc96ecb4640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.832+0000 7fc96ecb4640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc3137fe640 10 client.4278.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:49.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc3137fe640 15 client.4278.objecter _session_command_op_assign -1 1 2024-05-15T12:32:49.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc3137fe640 20 client.4278.objecter _get_session osd=-1 returning homeless 2024-05-15T12:32:49.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc3137fe640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc3137fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:49.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc3137fe640 1 -- 172.21.15.49:0/94651508 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc300002910 con 0x7fc3280af260 2024-05-15T12:32:49.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc96dcb2640 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2024-05-15T12:32:49.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc96dcb2640 1 -- 172.21.15.49:0/650259672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc960006e80 msgr2=0x7fc9600db950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc96dcb2640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc960006e80 0x7fc9600db950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc96dcb2640 1 -- 172.21.15.49:0/650259672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc960032180 msgr2=0x7fc96017f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc96dcb2640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc960032180 0x7fc96017f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc96dcb2640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc96dcb2640 1 monclient: found mon.noname-b 2024-05-15T12:32:49.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc96dcb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc96dcb2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.833+0000 7fc96dcb2640 1 -- 172.21.15.49:0/650259672 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600dbf20 con 0x7fc960030d40 2024-05-15T12:32:49.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.835+0000 7f2e36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.835+0000 7f2e36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.835+0000 7f2e36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.837+0000 7f2e37fff640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-05-15T12:32:49.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.837+0000 7f2e37fff640 1 -- 172.21.15.49:0/2760587583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30034c20 msgr2=0x7f2e300cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.837+0000 7f2e37fff640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e30034c20 0x7f2e300cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.837+0000 7f2e37fff640 1 -- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e3002fc10 msgr2=0x7f2e300cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.837+0000 7f2e37fff640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e3002fc10 0x7f2e300cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.837+0000 7f2e37fff640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.837+0000 7f2e37fff640 1 monclient: found mon.noname-c 2024-05-15T12:32:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.837+0000 7f2e37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.837+0000 7f2e37fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.837+0000 7f2e37fff640 1 -- 172.21.15.49:0/2760587583 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e300d0370 con 0x7f2e300cf5c0 2024-05-15T12:32:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.840+0000 7fc96dcb2640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc960030d40 0x7fc9600d6360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc968066a80 tx=0x7fc96806a350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.840+0000 7fc95effd640 10 client.?.objecter ms_handle_connect 0x7fc960030d40 2024-05-15T12:32:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.840+0000 7fc95effd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:49.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.840+0000 7ff57cd30640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5700c1590 0x7ff5700c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.840+0000 7ff57cd30640 10 monclient(hunting): get_auth_request con 0x7ff5700c1590 auth_method 0 2024-05-15T12:32:49.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.840+0000 7ff57cd30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:49.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.840+0000 7ff57cd30640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:49.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.840+0000 7ff57cd30640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:49.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.842+0000 7ff57cd30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:49.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.842+0000 7ff57cd30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:49.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.842+0000 7ff57cd30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:49.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.843+0000 7f2e37fff640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e300cf5c0 0x7f2e300cb490 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e38066a80 tx=0x7f2e3806a350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.843+0000 7f2e23fff640 10 client.?.objecter ms_handle_connect 0x7f2e300cf5c0 2024-05-15T12:32:49.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.843+0000 7f2e23fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.843+0000 7ff5777fe640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5700c1120 0x7ff57002b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.846+0000 7ff57cd30640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.846+0000 7ff57cd30640 1 -- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57000a960 msgr2=0x7ff5700c0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.846+0000 7ff57cd30640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57000a960 0x7ff5700c0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.846+0000 7ff57cd30640 1 -- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5700c1120 msgr2=0x7ff57002b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.846+0000 7ff57cd30640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5700c1120 0x7ff57002b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.846+0000 7ff57cd30640 10 monclient: _finish_hunting 0 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.846+0000 7ff57cd30640 1 monclient: found mon.noname-b 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.846+0000 7ff57cd30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.846+0000 7ff57cd30640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.846+0000 7ff57cd30640 1 -- 172.21.15.49:0/2673634926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57002bb40 con 0x7ff5700c1590 2024-05-15T12:32:49.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 1 -- 172.21.15.49:0/650259672 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9680731c0 con 0x7fc960030d40 2024-05-15T12:32:49.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:49.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 10 monclient: dump: 2024-05-15T12:32:49.941 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.941 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.941 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 1 -- 172.21.15.49:0/650259672 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96807f020 con 0x7fc960030d40 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 1 -- 172.21.15.49:0/650259672 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96806c110 con 0x7fc960030d40 2024-05-15T12:32:49.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc95effd640 10 monclient: dump: 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc96ccb0640 4 set_mon_vals no callback set 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc96f4b5640 5 monclient: authenticate success, global_id 4312 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc96f4b5640 10 monclient: _renew_subs 2024-05-15T12:32:49.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc96f4b5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.847+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96017f960 con 0x7fc960030d40 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 10 monclient: _renew_subs 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9601823d0 con 0x7fc960030d40 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 1 librados: init done 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 10 client.4312.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 15 client.4312.objecter _session_command_op_assign -1 1 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 20 client.4312.objecter _get_session osd=-1 returning homeless 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.848+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc9600d68a0 con 0x7fc960030d40 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.849+0000 7ff57cd30640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5700c1590 0x7ff5700c3240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff57806aa70 tx=0x7ff578068a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.849+0000 7ff574ff9640 10 client.?.objecter ms_handle_connect 0x7ff5700c1590 2024-05-15T12:32:49.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.849+0000 7ff574ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:49.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.857+0000 7ff33dffb640 1 -- 172.21.15.49:0/2670635754 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7ff344018020 con 0x7ff3480a9520 2024-05-15T12:32:49.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.858+0000 7ff33dffb640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7ff32c06cdd0 0x7ff32c06f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.858+0000 7ff34c9a7640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7ff32c06cdd0 0x7ff32c06f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.858+0000 7ff34c9a7640 10 monclient: get_auth_request con 0x7ff32c06cdd0 auth_method 0 2024-05-15T12:32:49.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.859+0000 7ff34c9a7640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7ff32c06cdd0 0x7ff32c06f290 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff348145ba0 tx=0x7ff338006c90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.859+0000 7ff33dffb640 10 client.4275.objecter ms_handle_connect 0x7ff32c06cdd0 2024-05-15T12:32:49.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.866+0000 7f4d8e7fc640 1 -- 172.21.15.49:0/566425273 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f4d9404b060 con 0x7f4d9806ef10 2024-05-15T12:32:49.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.866+0000 7f4d8e7fc640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4d7c06ce20 0x7f4d7c06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.866+0000 7f4d9d335640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4d7c06ce20 0x7f4d7c06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.866+0000 7f4d9d335640 10 monclient: get_auth_request con 0x7f4d7c06ce20 auth_method 0 2024-05-15T12:32:49.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.867+0000 7f4d9d335640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4d7c06ce20 0x7f4d7c06f2e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4d981d1a40 tx=0x7f4d88008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.867+0000 7f4d8e7fc640 10 client.4250.objecter ms_handle_connect 0x7f4d7c06ce20 2024-05-15T12:32:49.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 1 -- 172.21.15.49:0/2670635754 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff344088cb0 con 0x7ff3480a9520 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 10 client.4275.objecter ms_dispatch 0x7ff348000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 3 client.4275.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 3 client.4275.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 10 client.4275.objecter checking command 1 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] conn(0x7ff32c073210 0x7ff32c075630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 20 client.4275.objecter _get_session s=0x7ff32c072b40 osd=6 3 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 20 client.4275.objecter put_session s=0x7ff32c072b40 osd=6 3 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 15 client.4275.objecter _session_command_op_remove -1 1 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 20 client.4275.objecter _get_session s=0x7ff32c072b40 osd=6 3 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 20 client.4275.objecter get_session s=0x7ff32c072b40 osd=6 3 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 15 client.4275.objecter _session_command_op_assign 6 1 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 20 client.4275.objecter put_session s=0x7ff32c072b40 osd=6 4 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 10 client.4275.objecter _send_command 1 2024-05-15T12:32:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 1 -- 172.21.15.49:0/2670635754 --> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff344017e50 con 0x7ff32c073210 2024-05-15T12:32:49.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 20 client.4275.objecter dump_active .. 0 homeless 2024-05-15T12:32:49.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 1 -- 172.21.15.49:0/2670635754 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff344055170 con 0x7ff3480a9520 2024-05-15T12:32:49.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7ff33dffb640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:32:49.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.869+0000 7fc95effd640 1 -- 172.21.15.49:0/650259672 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fc96806c2d0 con 0x7fc960030d40 2024-05-15T12:32:49.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.870+0000 7fc95effd640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc94c06ce20 0x7fc94c06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.870+0000 7fc96e4b3640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc94c06ce20 0x7fc94c06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.870+0000 7fc96e4b3640 10 monclient: get_auth_request con 0x7fc94c06ce20 auth_method 0 2024-05-15T12:32:49.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.870+0000 7ff34d9a9640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] conn(0x7ff32c073210 0x7ff32c075630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.870+0000 7ff34d9a9640 10 monclient: get_auth_request con 0x7ff32c073210 auth_method 0 2024-05-15T12:32:49.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.870+0000 7fc96e4b3640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc94c06ce20 0x7fc94c06f2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc958036000 tx=0x7fc958031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.870+0000 7fc95effd640 10 client.4312.objecter ms_handle_connect 0x7fc94c06ce20 2024-05-15T12:32:49.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 1 -- 172.21.15.49:0/650259672 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc9680c0790 con 0x7fc960030d40 2024-05-15T12:32:49.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 10 client.4312.objecter ms_dispatch 0x7fc968000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:49.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 3 client.4312.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:49.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 3 client.4312.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:49.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 10 client.4312.objecter checking command 1 2024-05-15T12:32:49.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] conn(0x7fc94c073260 0x7fc94c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 20 client.4312.objecter _get_session s=0x7fc94c072b90 osd=4 3 2024-05-15T12:32:49.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 20 client.4312.objecter put_session s=0x7fc94c072b90 osd=4 3 2024-05-15T12:32:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 15 client.4312.objecter _session_command_op_remove -1 1 2024-05-15T12:32:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 20 client.4312.objecter _get_session s=0x7fc94c072b90 osd=4 3 2024-05-15T12:32:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 20 client.4312.objecter get_session s=0x7fc94c072b90 osd=4 3 2024-05-15T12:32:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 15 client.4312.objecter _session_command_op_assign 4 1 2024-05-15T12:32:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 20 client.4312.objecter put_session s=0x7fc94c072b90 osd=4 4 2024-05-15T12:32:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 10 client.4312.objecter _send_command 1 2024-05-15T12:32:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 1 -- 172.21.15.49:0/650259672 --> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc94c075d30 con 0x7fc94c073260 2024-05-15T12:32:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.872+0000 7fc95effd640 20 client.4312.objecter dump_active .. 0 homeless 2024-05-15T12:32:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.873+0000 7fc96ecb4640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] conn(0x7fc94c073260 0x7fc94c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fc96ecb4640 10 monclient: get_auth_request con 0x7fc94c073260 auth_method 0 2024-05-15T12:32:49.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 1 -- 172.21.15.49:0/566425273 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4d940792a0 con 0x7f4d9806ef10 2024-05-15T12:32:49.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 10 client.4250.objecter ms_dispatch 0x7f4d98000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:49.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 3 client.4250.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:49.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 3 client.4250.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:49.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 10 client.4250.objecter checking command 1 2024-05-15T12:32:49.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x7f4d7c0731d0 0x7f4d7c0755f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 20 client.4250.objecter _get_session s=0x7f4d7c072b00 osd=2 3 2024-05-15T12:32:49.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 20 client.4250.objecter put_session s=0x7f4d7c072b00 osd=2 3 2024-05-15T12:32:49.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 15 client.4250.objecter _session_command_op_remove -1 1 2024-05-15T12:32:49.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 20 client.4250.objecter _get_session s=0x7f4d7c072b00 osd=2 3 2024-05-15T12:32:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 20 client.4250.objecter get_session s=0x7f4d7c072b00 osd=2 3 2024-05-15T12:32:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 15 client.4250.objecter _session_command_op_assign 2 1 2024-05-15T12:32:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 20 client.4250.objecter put_session s=0x7f4d7c072b00 osd=2 4 2024-05-15T12:32:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 10 client.4250.objecter _send_command 1 2024-05-15T12:32:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 1 -- 172.21.15.49:0/566425273 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d94008f00 con 0x7f4d7c0731d0 2024-05-15T12:32:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 20 client.4250.objecter dump_active .. 0 homeless 2024-05-15T12:32:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 1 -- 172.21.15.49:0/566425273 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d940774a0 con 0x7f4d9806ef10 2024-05-15T12:32:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d8e7fc640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:32:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7ff33dffb640 10 client.4275.objecter ms_handle_connect 0x7ff32c073210 2024-05-15T12:32:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7ff34d9a9640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] conn(0x7ff32c073210 0x7ff32c075630 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fc95effd640 1 -- 172.21.15.49:0/650259672 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9680be990 con 0x7fc960030d40 2024-05-15T12:32:49.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fc95effd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:32:49.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd38affd640 1 -- 172.21.15.49:0/3521073038 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390017070 con 0x7fd394007070 2024-05-15T12:32:49.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd38affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd38affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:49.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd38affd640 10 monclient: dump: 2024-05-15T12:32:49.955 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.955 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.955 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd38affd640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd38affd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd38affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7f4d9e337640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x7f4d7c0731d0 0x7f4d7c0755f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd3a0e97640 5 monclient: authenticate success, global_id 4253 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd38affd640 1 -- 172.21.15.49:0/3521073038 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3900035f0 con 0x7fd394007070 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd3a0e97640 10 monclient: _renew_subs 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd3a0e97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.874+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd394030410 con 0x7fd394007070 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd38affd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd38affd640 1 -- 172.21.15.49:0/3521073038 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390017070 con 0x7fd394007070 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd38affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd38affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd38affd640 10 monclient: dump: 2024-05-15T12:32:49.958 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd398d99640 4 set_mon_vals no callback set 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd3a0e97640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd3a0e97640 10 monclient: _renew_subs 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd3a0e97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3940c73b0 con 0x7fd394007070 2024-05-15T12:32:49.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7fd3a0e97640 1 librados: init done 2024-05-15T12:32:49.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.875+0000 7f4d9e337640 10 monclient: get_auth_request con 0x7f4d7c0731d0 auth_method 0 2024-05-15T12:32:49.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 1 -- 172.21.15.49:0/862655462 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370017070 con 0x7f9388141b20 2024-05-15T12:32:49.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 10 monclient: dump: 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 1 -- 172.21.15.49:0/862655462 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93700035f0 con 0x7f9388141b20 2024-05-15T12:32:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 1 -- 172.21.15.49:0/862655462 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93700039e0 con 0x7f9388141b20 2024-05-15T12:32:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f937f7fe640 10 monclient: dump: 2024-05-15T12:32:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.962 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.963 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.963 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.963 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.963 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.963 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.963 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.963 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f938d1f4640 5 monclient: authenticate success, global_id 4303 2024-05-15T12:32:49.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f938d1f4640 10 monclient: _renew_subs 2024-05-15T12:32:49.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f938d1f4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938813eae0 con 0x7f9388141b20 2024-05-15T12:32:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f938d1f4640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f938d1f4640 10 monclient: _renew_subs 2024-05-15T12:32:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f938d1f4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93881d7610 con 0x7f9388141b20 2024-05-15T12:32:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f938d1f4640 1 librados: init done 2024-05-15T12:32:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f938559a640 4 set_mon_vals no callback set 2024-05-15T12:32:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7fd3a0e97640 10 client.4253.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7fd3a0e97640 15 client.4253.objecter _session_command_op_assign -1 1 2024-05-15T12:32:49.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7fd3a0e97640 20 client.4253.objecter _get_session osd=-1 returning homeless 2024-05-15T12:32:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7fd3a0e97640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7fd3a0e97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd394005710 con 0x7fd394007070 2024-05-15T12:32:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 1 -- 172.21.15.49:0/2760587583 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3804f650 con 0x7f2e300cf5c0 2024-05-15T12:32:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 10 monclient: dump: 2024-05-15T12:32:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 1 -- 172.21.15.49:0/2760587583 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e38090020 con 0x7f2e300cf5c0 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 1 -- 172.21.15.49:0/2760587583 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3806c110 con 0x7f2e300cf5c0 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e23fff640 10 monclient: dump: 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7f2e35ffb640 4 set_mon_vals no callback set 2024-05-15T12:32:49.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f938d1f4640 10 client.4303.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:49.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f938d1f4640 15 client.4303.objecter _session_command_op_assign -1 1 2024-05-15T12:32:49.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f938d1f4640 20 client.4303.objecter _get_session osd=-1 returning homeless 2024-05-15T12:32:49.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f938d1f4640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f938d1f4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f938804f0e0 con 0x7f9388141b20 2024-05-15T12:32:49.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f2e3cff7640 5 monclient: authenticate success, global_id 4259 2024-05-15T12:32:49.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f2e3cff7640 10 monclient: _renew_subs 2024-05-15T12:32:49.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f2e3cff7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f2e3cff7640 1 -- 172.21.15.49:0/2760587583 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e300caa40 con 0x7f2e300cf5c0 2024-05-15T12:32:49.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f2e3cff7640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f2e3cff7640 10 monclient: _renew_subs 2024-05-15T12:32:49.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f2e3cff7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f2e3cff7640 1 -- 172.21.15.49:0/2760587583 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e300c9e40 con 0x7f2e300cf5c0 2024-05-15T12:32:49.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7f2e3cff7640 1 librados: init done 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 1 -- 172.21.15.49:0/2673634926 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57806a6e0 con 0x7ff5700c1590 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 10 monclient: dump: 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.970 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 10 monclient: _finish_auth 0 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 10 monclient: _check_auth_tickets 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 1 -- 172.21.15.49:0/2673634926 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5780731c0 con 0x7ff5700c1590 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 1 -- 172.21.15.49:0/2673634926 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578093400 con 0x7ff5700c1590 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff574ff9640 10 monclient: dump: 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.971 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.876+0000 7ff5767fc640 4 set_mon_vals no callback set 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7ff57d531640 5 monclient: authenticate success, global_id 4318 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7ff57d531640 10 monclient: _renew_subs 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7ff57d531640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57002cc20 con 0x7ff5700c1590 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7ff57d531640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7ff57d531640 10 monclient: _renew_subs 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7ff57d531640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57002a830 con 0x7ff5700c1590 2024-05-15T12:32:49.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.877+0000 7ff57d531640 1 librados: init done 2024-05-15T12:32:49.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7fc96ecb4640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] conn(0x7fc94c073260 0x7fc94c075680 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7fc95effd640 10 client.4312.objecter ms_handle_connect 0x7fc94c073260 2024-05-15T12:32:49.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7ff57d531640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:49.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7ff57d531640 15 client.4318.objecter _session_command_op_assign -1 1 2024-05-15T12:32:49.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7ff57d531640 20 client.4318.objecter _get_session osd=-1 returning homeless 2024-05-15T12:32:49.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7f2e3cff7640 10 client.4259.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:49.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7ff57d531640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7f2e3cff7640 15 client.4259.objecter _session_command_op_assign -1 1 2024-05-15T12:32:49.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7ff57d531640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:49.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7f2e3cff7640 20 client.4259.objecter _get_session osd=-1 returning homeless 2024-05-15T12:32:49.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7f2e3cff7640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:49.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5700052c0 con 0x7ff5700c1590 2024-05-15T12:32:49.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7f2e3cff7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:49.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.878+0000 7f2e3cff7640 1 -- 172.21.15.49:0/2760587583 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2e300052c0 con 0x7f2e300cf5c0 2024-05-15T12:32:49.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.880+0000 7f4d9e337640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x7f4d7c0731d0 0x7f4d7c0755f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.880+0000 7f4d8e7fc640 10 client.4250.objecter ms_handle_connect 0x7f4d7c0731d0 2024-05-15T12:32:49.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.885+0000 7fc31dffb640 1 -- 172.21.15.49:0/94651508 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fc31801c460 con 0x7fc3280af260 2024-05-15T12:32:49.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.886+0000 7fc31dffb640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc30c06ce50 0x7fc30c06f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.886+0000 7fc32d2d9640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc30c06ce50 0x7fc30c06f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.886+0000 7fc32d2d9640 10 monclient: get_auth_request con 0x7fc30c06ce50 auth_method 0 2024-05-15T12:32:49.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.886+0000 7fc32d2d9640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc30c06ce50 0x7fc30c06f310 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc320035000 tx=0x7fc32002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.886+0000 7fc31dffb640 10 client.4278.objecter ms_handle_connect 0x7fc30c06ce50 2024-05-15T12:32:49.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.889+0000 7fc31dffb640 1 -- 172.21.15.49:0/94651508 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc318057340 con 0x7fc3280af260 2024-05-15T12:32:49.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.889+0000 7fc31dffb640 10 client.4278.objecter ms_dispatch 0x7fc328000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.889+0000 7fc31dffb640 3 client.4278.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.889+0000 7fc31dffb640 3 client.4278.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 10 client.4278.objecter checking command 1 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] conn(0x7fc30c073290 0x7fc30c0756b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 20 client.4278.objecter _get_session s=0x7fc30c072bc0 osd=1 3 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 20 client.4278.objecter put_session s=0x7fc30c072bc0 osd=1 3 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 15 client.4278.objecter _session_command_op_remove -1 1 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 20 client.4278.objecter _get_session s=0x7fc30c072bc0 osd=1 3 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 20 client.4278.objecter get_session s=0x7fc30c072bc0 osd=1 3 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 15 client.4278.objecter _session_command_op_assign 1 1 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 20 client.4278.objecter put_session s=0x7fc30c072bc0 osd=1 4 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 10 client.4278.objecter _send_command 1 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 1 -- 172.21.15.49:0/94651508 --> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc318023ec0 con 0x7fc30c073290 2024-05-15T12:32:49.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 20 client.4278.objecter dump_active .. 0 homeless 2024-05-15T12:32:49.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 1 -- 172.21.15.49:0/94651508 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc318055540 con 0x7fc3280af260 2024-05-15T12:32:49.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc31dffb640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:32:49.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.890+0000 7fc32cad8640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] conn(0x7fc30c073290 0x7fc30c0756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.891+0000 7fc32cad8640 10 monclient: get_auth_request con 0x7fc30c073290 auth_method 0 2024-05-15T12:32:49.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.895+0000 7fc31dffb640 10 client.4278.objecter ms_handle_connect 0x7fc30c073290 2024-05-15T12:32:49.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.895+0000 7fc32cad8640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] conn(0x7fc30c073290 0x7fc30c0756b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.900+0000 7fd38affd640 1 -- 172.21.15.49:0/3521073038 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fd390003800 con 0x7fd394007070 2024-05-15T12:32:49.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.901+0000 7fd38affd640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd37c06cfd0 0x7fd37c06f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.901+0000 7fd399d9b640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd37c06cfd0 0x7fd37c06f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.901+0000 7fd399d9b640 10 monclient: get_auth_request con 0x7fd37c06cfd0 auth_method 0 2024-05-15T12:32:49.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.901+0000 7fd399d9b640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd37c06cfd0 0x7fd37c06f490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3940c88e0 tx=0x7fd38c0091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.901+0000 7fd38affd640 10 client.4253.objecter ms_handle_connect 0x7fd37c06cfd0 2024-05-15T12:32:49.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.902+0000 7f937f7fe640 1 -- 172.21.15.49:0/862655462 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f937001a070 con 0x7f9388141b20 2024-05-15T12:32:49.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.903+0000 7f937f7fe640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f9368071230 0x7f93680736f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.903+0000 7f9386d9d640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f9368071230 0x7f93680736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.903+0000 7f9386d9d640 10 monclient: get_auth_request con 0x7f9368071230 auth_method 0 2024-05-15T12:32:49.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7f9386d9d640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f9368071230 0x7f93680736f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9380009710 tx=0x7f9380006c90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7f937f7fe640 10 client.4303.objecter ms_handle_connect 0x7f9368071230 2024-05-15T12:32:49.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 1 -- 172.21.15.49:0/3521073038 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd390087a20 con 0x7fd394007070 2024-05-15T12:32:49.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 10 client.4253.objecter ms_dispatch 0x7fd39c000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:49.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 3 client.4253.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:49.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 3 client.4253.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:49.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 10 client.4253.objecter checking command 1 2024-05-15T12:32:49.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] conn(0x7fd37c073410 0x7fd37c075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 20 client.4253.objecter _get_session s=0x7fd37c072d40 osd=5 3 2024-05-15T12:32:49.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 20 client.4253.objecter put_session s=0x7fd37c072d40 osd=5 3 2024-05-15T12:32:49.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 15 client.4253.objecter _session_command_op_remove -1 1 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 20 client.4253.objecter _get_session s=0x7fd37c072d40 osd=5 3 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 20 client.4253.objecter get_session s=0x7fd37c072d40 osd=5 3 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 15 client.4253.objecter _session_command_op_assign 5 1 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 20 client.4253.objecter put_session s=0x7fd37c072d40 osd=5 4 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 10 client.4253.objecter _send_command 1 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 1 -- 172.21.15.49:0/3521073038 --> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd390009e50 con 0x7fd37c073410 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 20 client.4253.objecter dump_active .. 0 homeless 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 1 -- 172.21.15.49:0/3521073038 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd390053fd0 con 0x7fd394007070 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.904+0000 7fd38affd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.905+0000 7fd39ad9d640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] conn(0x7fd37c073410 0x7fd37c075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.905+0000 7fd39ad9d640 10 monclient: get_auth_request con 0x7fd37c073410 auth_method 0 2024-05-15T12:32:49.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 1 -- 172.21.15.49:0/862655462 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9370056560 con 0x7f9388141b20 2024-05-15T12:32:49.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 10 client.4303.objecter ms_dispatch 0x7f9388000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:49.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 3 client.4303.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:49.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 3 client.4303.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:49.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 10 client.4303.objecter checking command 1 2024-05-15T12:32:49.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] conn(0x7f9368077670 0x7f9368079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 20 client.4303.objecter _get_session s=0x7f9368076fa0 osd=7 3 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 20 client.4303.objecter put_session s=0x7f9368076fa0 osd=7 3 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 15 client.4303.objecter _session_command_op_remove -1 1 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 20 client.4303.objecter _get_session s=0x7f9368076fa0 osd=7 3 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 20 client.4303.objecter get_session s=0x7f9368076fa0 osd=7 3 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 15 client.4303.objecter _session_command_op_assign 7 1 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 20 client.4303.objecter put_session s=0x7f9368076fa0 osd=7 4 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 10 client.4303.objecter _send_command 1 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 1 -- 172.21.15.49:0/862655462 --> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f936807a140 con 0x7f9368077670 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.908+0000 7f937f7fe640 20 client.4303.objecter dump_active .. 0 homeless 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.909+0000 7f938659c640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] conn(0x7f9368077670 0x7f9368079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.909+0000 7f938659c640 10 monclient: get_auth_request con 0x7f9368077670 auth_method 0 2024-05-15T12:32:49.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.910+0000 7fd39ad9d640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] conn(0x7fd37c073410 0x7fd37c075830 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.910+0000 7fd38affd640 10 client.4253.objecter ms_handle_connect 0x7fd37c073410 2024-05-15T12:32:49.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.913+0000 7f937f7fe640 10 client.4303.objecter ms_handle_connect 0x7f9368077670 2024-05-15T12:32:49.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.913+0000 7f938659c640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] conn(0x7f9368077670 0x7f9368079a90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.924+0000 7f2e23fff640 1 -- 172.21.15.49:0/2760587583 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f2e3806c2b0 con 0x7f2e300cf5c0 2024-05-15T12:32:49.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.925+0000 7f2e23fff640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2e1c06ce20 0x7f2e1c06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.925+0000 7f2e377fe640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2e1c06ce20 0x7f2e1c06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.925+0000 7f2e377fe640 10 monclient: get_auth_request con 0x7f2e1c06ce20 auth_method 0 2024-05-15T12:32:49.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.926+0000 7f2e23fff640 10 client.4259.objecter ms_handle_connect 0x7f2e1c06ce20 2024-05-15T12:32:49.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.926+0000 7f2e377fe640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2e1c06ce20 0x7f2e1c06f2e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2e24036000 tx=0x7f2e24030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.926+0000 7ff574ff9640 1 -- 172.21.15.49:0/2673634926 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7ff5780935a0 con 0x7ff5700c1590 2024-05-15T12:32:49.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.926+0000 7ff574ff9640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7ff55c06ce10 0x7ff55c06f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 1 -- 172.21.15.49:0/2760587583 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2e38103330 con 0x7f2e300cf5c0 2024-05-15T12:32:49.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 10 client.4259.objecter ms_dispatch 0x7f2e38000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:49.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 3 client.4259.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:49.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 3 client.4259.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:49.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 10 client.4259.objecter checking command 1 2024-05-15T12:32:49.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] conn(0x7f2e1c073260 0x7f2e1c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 20 client.4259.objecter _get_session s=0x7f2e1c072b90 osd=0 3 2024-05-15T12:32:49.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 20 client.4259.objecter put_session s=0x7f2e1c072b90 osd=0 3 2024-05-15T12:32:49.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 15 client.4259.objecter _session_command_op_remove -1 1 2024-05-15T12:32:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 20 client.4259.objecter _get_session s=0x7f2e1c072b90 osd=0 3 2024-05-15T12:32:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 20 client.4259.objecter get_session s=0x7f2e1c072b90 osd=0 3 2024-05-15T12:32:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 15 client.4259.objecter _session_command_op_assign 0 1 2024-05-15T12:32:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 20 client.4259.objecter put_session s=0x7f2e1c072b90 osd=0 4 2024-05-15T12:32:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 10 client.4259.objecter _send_command 1 2024-05-15T12:32:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 1 -- 172.21.15.49:0/2760587583 --> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e38091ec0 con 0x7f2e1c073260 2024-05-15T12:32:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 20 client.4259.objecter dump_active .. 0 homeless 2024-05-15T12:32:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 1 -- 172.21.15.49:0/2760587583 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e380cf7f0 con 0x7f2e300cf5c0 2024-05-15T12:32:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e23fff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:32:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7ff574ff9640 1 -- 172.21.15.49:0/2673634926 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff5780d13a0 con 0x7ff5700c1590 2024-05-15T12:32:49.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7ff574ff9640 10 client.4318.objecter ms_dispatch 0x7ff578000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:49.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7ff574ff9640 3 client.4318.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:49.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7ff574ff9640 3 client.4318.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:49.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.930+0000 7f2e36ffd640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] conn(0x7f2e1c073260 0x7f2e1c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 10 client.4318.objecter checking command 1 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] conn(0x7ff55c0731c0 0x7ff55c0755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 20 client.4318.objecter _get_session s=0x7ff55c072af0 osd=3 3 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 20 client.4318.objecter put_session s=0x7ff55c072af0 osd=3 3 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 15 client.4318.objecter _session_command_op_remove -1 1 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 20 client.4318.objecter _get_session s=0x7ff55c072af0 osd=3 3 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 20 client.4318.objecter get_session s=0x7ff55c072af0 osd=3 3 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 15 client.4318.objecter _session_command_op_assign 3 1 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 20 client.4318.objecter put_session s=0x7ff55c072af0 osd=3 4 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 10 client.4318.objecter _send_command 1 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 1 -- 172.21.15.49:0/2673634926 --> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff578094e50 con 0x7ff55c0731c0 2024-05-15T12:32:49.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 20 client.4318.objecter dump_active .. 0 homeless 2024-05-15T12:32:49.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7f2e36ffd640 10 monclient: get_auth_request con 0x7f2e1c073260 auth_method 0 2024-05-15T12:32:49.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 1 -- 172.21.15.49:0/2673634926 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5780cf5a0 con 0x7ff5700c1590 2024-05-15T12:32:49.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff574ff9640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:32:49.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff5777fe640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] conn(0x7ff55c0731c0 0x7ff55c0755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.931+0000 7ff5777fe640 10 monclient: get_auth_request con 0x7ff55c0731c0 auth_method 0 2024-05-15T12:32:49.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.933+0000 7f937f7fe640 1 -- 172.21.15.49:0/862655462 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9370054760 con 0x7f9388141b20 2024-05-15T12:32:49.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.933+0000 7f937f7fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:32:49.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.934+0000 7f2e36ffd640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] conn(0x7f2e1c073260 0x7f2e1c075680 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.934+0000 7f2e23fff640 10 client.4259.objecter ms_handle_connect 0x7f2e1c073260 2024-05-15T12:32:49.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.934+0000 7ff574ff9640 10 client.4318.objecter ms_handle_connect 0x7ff55c0731c0 2024-05-15T12:32:49.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.934+0000 7ff5777fe640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] conn(0x7ff55c0731c0 0x7ff55c0755e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.935+0000 7ff577fff640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7ff55c06ce10 0x7ff55c06f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:49.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.935+0000 7ff577fff640 10 monclient: get_auth_request con 0x7ff55c06ce10 auth_method 0 2024-05-15T12:32:49.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.935+0000 7ff574ff9640 10 client.4318.objecter ms_handle_connect 0x7ff55c06ce10 2024-05-15T12:32:49.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.935+0000 7ff577fff640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7ff55c06ce10 0x7ff55c06f2d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff56c036040 tx=0x7ff56c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:49.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.948+0000 7fc95effd640 1 -- 172.21.15.49:0/650259672 <== osd.4 v2:172.21.15.88:6816/3384996234 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc94c075d30 con 0x7fc94c073260 2024-05-15T12:32:49.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.948+0000 7fc95effd640 10 client.4312.objecter ms_dispatch 0x7fc968000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:32:49.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.948+0000 7fc95effd640 10 client.4312.objecter _finish_command 1 = system:0 2024-05-15T12:32:49.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.948+0000 7fc95effd640 20 client.4312.objecter put_session s=0x7fc94c072b90 osd=4 4 2024-05-15T12:32:49.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.948+0000 7fc95effd640 15 client.4312.objecter _session_command_op_remove 4 1 2024-05-15T12:32:49.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.949+0000 7ff33dffb640 1 -- 172.21.15.49:0/2670635754 <== osd.6 v2:172.21.15.88:6800/3138490191 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff344017e50 con 0x7ff32c073210 2024-05-15T12:32:49.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.949+0000 7ff33dffb640 10 client.4275.objecter ms_dispatch 0x7ff348000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:32:49.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.949+0000 7ff33dffb640 10 client.4275.objecter _finish_command 1 = system:0 2024-05-15T12:32:49.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.949+0000 7ff33dffb640 20 client.4275.objecter put_session s=0x7ff32c072b40 osd=6 4 2024-05-15T12:32:49.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.949+0000 7ff33dffb640 15 client.4275.objecter _session_command_op_remove 6 1 2024-05-15T12:32:49.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 10 client.4312.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:32:49.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 15 client.4312.objecter _session_command_op_assign -1 2 2024-05-15T12:32:49.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 20 client.4312.objecter _get_session s=0x7fc94c072b90 osd=4 3 2024-05-15T12:32:49.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 20 client.4312.objecter put_session s=0x7fc94c072b90 osd=4 3 2024-05-15T12:32:49.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 20 client.4312.objecter _get_session s=0x7fc94c072b90 osd=4 3 2024-05-15T12:32:49.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 15 client.4312.objecter _session_command_op_remove -1 2 2024-05-15T12:32:49.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 20 client.4312.objecter get_session s=0x7fc94c072b90 osd=4 3 2024-05-15T12:32:49.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 15 client.4312.objecter _session_command_op_assign 4 2 2024-05-15T12:32:49.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 20 client.4312.objecter put_session s=0x7fc94c072b90 osd=4 4 2024-05-15T12:32:49.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 10 client.4312.objecter _send_command 2 2024-05-15T12:32:49.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.962+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 --> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc96017f1e0 con 0x7fc94c073260 2024-05-15T12:32:49.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.970+0000 7f4d8e7fc640 1 -- 172.21.15.49:0/566425273 <== osd.2 v2:172.21.15.49:6826/2361777220 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4d94008f00 con 0x7f4d7c0731d0 2024-05-15T12:32:49.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.970+0000 7f4d8e7fc640 10 client.4250.objecter ms_dispatch 0x7f4d98000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:32:49.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.970+0000 7f4d8e7fc640 10 client.4250.objecter _finish_command 1 = system:0 2024-05-15T12:32:49.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.970+0000 7f4d8e7fc640 20 client.4250.objecter put_session s=0x7f4d7c072b00 osd=2 4 2024-05-15T12:32:49.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.970+0000 7f4d8e7fc640 15 client.4250.objecter _session_command_op_remove 2 1 2024-05-15T12:32:49.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc95effd640 1 -- 172.21.15.49:0/650259672 <== osd.4 v2:172.21.15.88:6816/3384996234 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc96017f1e0 con 0x7fc94c073260 2024-05-15T12:32:49.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc95effd640 10 client.4312.objecter ms_dispatch 0x7fc968000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:32:49.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc95effd640 10 client.4312.objecter _finish_command 2 = system:0 2024-05-15T12:32:49.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc95effd640 20 client.4312.objecter put_session s=0x7fc94c072b90 osd=4 4 2024-05-15T12:32:49.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc95effd640 15 client.4312.objecter _session_command_op_remove 4 2 2024-05-15T12:32:50.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 10 librados: watch_flush enter 2024-05-15T12:32:50.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 10 librados: watch_flush exit 2024-05-15T12:32:50.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 10 client.4312.objecter close_session for osd.4 2024-05-15T12:32:50.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 >> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] conn(0x7fc94c073260 msgr2=0x7fc94c075680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] conn(0x7fc94c073260 0x7fc94c075680 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 20 client.4312.objecter put_session s=0x7fc94c072b90 osd=4 1 2024-05-15T12:32:50.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 20 client.4312.objecter shutdown clearing up homeless session... 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 10 client.4312.objecter successfully canceled tick 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc94c06ce20 msgr2=0x7fc94c06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc94c06ce20 0x7fc94c06f2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc958036000 tx=0x7fc958031000 comp rx=0 tx=0).stop 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 10 monclient: shutdown 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc960030d40 msgr2=0x7fc9600d6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 1 --2- 172.21.15.49:0/650259672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc960030d40 0x7fc9600d6360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc968066a80 tx=0x7fc96806a350 comp rx=0 tx=0).stop 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96ecb4640 1 -- 172.21.15.49:0/650259672 reap_dead start 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 shutdown_connections 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 >> 172.21.15.49:0/650259672 conn(0x7fc960005ee0 msgr2=0x7fc9600e1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 shutdown_connections 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.974+0000 7fc96f4b5640 1 -- 172.21.15.49:0/650259672 wait complete. 2024-05-15T12:32:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.975+0000 7fc96f4b5640 1 librados: shutdown 2024-05-15T12:32:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.980+0000 7ff34e1aa640 10 client.4275.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:32:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.980+0000 7ff34e1aa640 15 client.4275.objecter _session_command_op_assign -1 2 2024-05-15T12:32:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.980+0000 7ff34e1aa640 20 client.4275.objecter _get_session s=0x7ff32c072b40 osd=6 3 2024-05-15T12:32:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.980+0000 7ff34e1aa640 20 client.4275.objecter put_session s=0x7ff32c072b40 osd=6 3 2024-05-15T12:32:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.980+0000 7ff34e1aa640 20 client.4275.objecter _get_session s=0x7ff32c072b40 osd=6 3 2024-05-15T12:32:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.980+0000 7ff34e1aa640 15 client.4275.objecter _session_command_op_remove -1 2 2024-05-15T12:32:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.980+0000 7ff34e1aa640 20 client.4275.objecter get_session s=0x7ff32c072b40 osd=6 3 2024-05-15T12:32:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.981+0000 7ff34e1aa640 15 client.4275.objecter _session_command_op_assign 6 2 2024-05-15T12:32:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.981+0000 7ff34e1aa640 20 client.4275.objecter put_session s=0x7ff32c072b40 osd=6 4 2024-05-15T12:32:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.981+0000 7ff34e1aa640 10 client.4275.objecter _send_command 2 2024-05-15T12:32:50.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.981+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 --> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff348146630 con 0x7ff32c073210 2024-05-15T12:32:50.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.984+0000 7fc31dffb640 1 -- 172.21.15.49:0/94651508 <== osd.1 v2:172.21.15.49:6810/3250791401 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc318023ec0 con 0x7fc30c073290 2024-05-15T12:32:50.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.984+0000 7fc31dffb640 10 client.4278.objecter ms_dispatch 0x7fc328000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:32:50.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.984+0000 7fc31dffb640 10 client.4278.objecter _finish_command 1 = system:0 2024-05-15T12:32:50.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.985+0000 7fc31dffb640 20 client.4278.objecter put_session s=0x7fc30c072bc0 osd=1 4 2024-05-15T12:32:50.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.985+0000 7fc31dffb640 15 client.4278.objecter _session_command_op_remove 1 1 2024-05-15T12:32:50.004 INFO:teuthology.orchestra.run.smithi049.stdout:55834574854 2024-05-15T12:32:50.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-15T12:32:50.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.991+0000 7ff33dffb640 1 -- 172.21.15.49:0/2670635754 <== osd.6 v2:172.21.15.88:6800/3138490191 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff34002c7e0 con 0x7ff32c073210 2024-05-15T12:32:50.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.991+0000 7ff33dffb640 10 client.4275.objecter ms_dispatch 0x7ff348000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:32:50.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.991+0000 7ff33dffb640 10 client.4275.objecter _finish_command 2 = system:0 2024-05-15T12:32:50.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.991+0000 7ff33dffb640 20 client.4275.objecter put_session s=0x7ff32c072b40 osd=6 4 2024-05-15T12:32:50.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.991+0000 7ff33dffb640 15 client.4275.objecter _session_command_op_remove 6 2 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 10 librados: watch_flush enter 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 10 librados: watch_flush exit 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 10 client.4275.objecter close_session for osd.6 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 >> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] conn(0x7ff32c073210 msgr2=0x7ff32c075630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] conn(0x7ff32c073210 0x7ff32c075630 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 20 client.4275.objecter put_session s=0x7ff32c072b40 osd=6 1 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 20 client.4275.objecter shutdown clearing up homeless session... 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 10 client.4275.objecter successfully canceled tick 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7ff32c06cdd0 msgr2=0x7ff32c06f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7ff32c06cdd0 0x7ff32c06f290 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff348145ba0 tx=0x7ff338006c90 comp rx=0 tx=0).stop 2024-05-15T12:32:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 10 monclient: shutdown 2024-05-15T12:32:50.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3480a9520 msgr2=0x7ff348144450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 1 --2- 172.21.15.49:0/2670635754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3480a9520 0x7ff348144450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff34400b3e0 tx=0x7ff34400b8a0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34d9a9640 1 -- 172.21.15.49:0/2670635754 reap_dead start 2024-05-15T12:32:50.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 shutdown_connections 2024-05-15T12:32:50.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 >> 172.21.15.49:0/2670635754 conn(0x7ff34806c110 msgr2=0x7ff34806c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.992+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 shutdown_connections 2024-05-15T12:32:50.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.993+0000 7ff34e1aa640 1 -- 172.21.15.49:0/2670635754 wait complete. 2024-05-15T12:32:50.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.993+0000 7ff34e1aa640 1 librados: shutdown 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 10 client.4250.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 15 client.4250.objecter _session_command_op_assign -1 2 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 20 client.4250.objecter _get_session s=0x7f4d7c072b00 osd=2 3 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 20 client.4250.objecter put_session s=0x7f4d7c072b00 osd=2 3 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 20 client.4250.objecter _get_session s=0x7f4d7c072b00 osd=2 3 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 15 client.4250.objecter _session_command_op_remove -1 2 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 20 client.4250.objecter get_session s=0x7f4d7c072b00 osd=2 3 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 15 client.4250.objecter _session_command_op_assign 2 2 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 20 client.4250.objecter put_session s=0x7f4d7c072b00 osd=2 4 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 10 client.4250.objecter _send_command 2 2024-05-15T12:32:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:49.999+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4d9804f0e0 con 0x7f4d7c0731d0 2024-05-15T12:32:50.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.005+0000 7fd38affd640 1 -- 172.21.15.49:0/3521073038 <== osd.5 v2:172.21.15.88:6808/3085185190 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd390009e50 con 0x7fd37c073410 2024-05-15T12:32:50.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.005+0000 7fd38affd640 10 client.4253.objecter ms_dispatch 0x7fd39c000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:32:50.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.005+0000 7fd38affd640 10 client.4253.objecter _finish_command 1 = system:0 2024-05-15T12:32:50.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.005+0000 7fd38affd640 20 client.4253.objecter put_session s=0x7fd37c072d40 osd=5 4 2024-05-15T12:32:50.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.005+0000 7fd38affd640 15 client.4253.objecter _session_command_op_remove 5 1 2024-05-15T12:32:50.011 INFO:teuthology.orchestra.run.smithi049.stdout:60129542153 2024-05-15T12:32:50.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-15T12:32:50.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.009+0000 7f2e23fff640 1 -- 172.21.15.49:0/2760587583 <== osd.0 v2:172.21.15.49:6818/3376151072 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2e38091ec0 con 0x7f2e1c073260 2024-05-15T12:32:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.009+0000 7f2e23fff640 10 client.4259.objecter ms_dispatch 0x7f2e38000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:32:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.009+0000 7f2e23fff640 10 client.4259.objecter _finish_command 1 = system:0 2024-05-15T12:32:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.009+0000 7f2e23fff640 20 client.4259.objecter put_session s=0x7f2e1c072b90 osd=0 4 2024-05-15T12:32:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.009+0000 7f2e23fff640 15 client.4259.objecter _session_command_op_remove 0 1 2024-05-15T12:32:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 10 client.4278.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:32:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 15 client.4278.objecter _session_command_op_assign -1 2 2024-05-15T12:32:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 20 client.4278.objecter _get_session s=0x7fc30c072bc0 osd=1 3 2024-05-15T12:32:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 20 client.4278.objecter put_session s=0x7fc30c072bc0 osd=1 3 2024-05-15T12:32:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 20 client.4278.objecter _get_session s=0x7fc30c072bc0 osd=1 3 2024-05-15T12:32:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 15 client.4278.objecter _session_command_op_remove -1 2 2024-05-15T12:32:50.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 20 client.4278.objecter get_session s=0x7fc30c072bc0 osd=1 3 2024-05-15T12:32:50.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 15 client.4278.objecter _session_command_op_assign 1 2 2024-05-15T12:32:50.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 20 client.4278.objecter put_session s=0x7fc30c072bc0 osd=1 4 2024-05-15T12:32:50.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 10 client.4278.objecter _send_command 2 2024-05-15T12:32:50.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.013+0000 7fc3137fe640 1 -- 172.21.15.49:0/94651508 --> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc300001240 con 0x7fc30c073290 2024-05-15T12:32:50.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.014+0000 7f937f7fe640 1 -- 172.21.15.49:0/862655462 <== osd.7 v2:172.21.15.88:6824/2037618931 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f936807a140 con 0x7f9368077670 2024-05-15T12:32:50.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.014+0000 7f937f7fe640 10 client.4303.objecter ms_dispatch 0x7f9388000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:32:50.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.014+0000 7f937f7fe640 10 client.4303.objecter _finish_command 1 = system:0 2024-05-15T12:32:50.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.014+0000 7f937f7fe640 20 client.4303.objecter put_session s=0x7f9368076fa0 osd=7 4 2024-05-15T12:32:50.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.014+0000 7f937f7fe640 15 client.4303.objecter _session_command_op_remove 7 1 2024-05-15T12:32:50.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.017+0000 7f4d8e7fc640 1 -- 172.21.15.49:0/566425273 <== osd.2 v2:172.21.15.49:6826/2361777220 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4d9804f0e0 con 0x7f4d7c0731d0 2024-05-15T12:32:50.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.017+0000 7f4d8e7fc640 10 client.4250.objecter ms_dispatch 0x7f4d98000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:32:50.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.017+0000 7f4d8e7fc640 10 client.4250.objecter _finish_command 2 = system:0 2024-05-15T12:32:50.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.017+0000 7f4d8e7fc640 20 client.4250.objecter put_session s=0x7f4d7c072b00 osd=2 4 2024-05-15T12:32:50.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.017+0000 7f4d8e7fc640 15 client.4250.objecter _session_command_op_remove 2 2 2024-05-15T12:32:50.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.017+0000 7f4d9fd9a640 10 librados: watch_flush enter 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 10 librados: watch_flush exit 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 10 client.4250.objecter close_session for osd.2 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x7f4d7c0731d0 msgr2=0x7f4d7c0755f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x7f4d7c0731d0 0x7f4d7c0755f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 20 client.4250.objecter put_session s=0x7f4d7c072b00 osd=2 1 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 20 client.4250.objecter shutdown clearing up homeless session... 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 10 client.4250.objecter successfully canceled tick 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4d7c06ce20 msgr2=0x7f4d7c06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4d7c06ce20 0x7f4d7c06f2e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4d981d1a40 tx=0x7f4d88008040 comp rx=0 tx=0).stop 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 10 monclient: shutdown 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d9806ef10 msgr2=0x7f4d98149cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9fd9a640 1 --2- 172.21.15.49:0/566425273 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d9806ef10 0x7f4d98149cb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4d94000c00 tx=0x7f4d940046c0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.020+0000 7f4d9e337640 1 -- 172.21.15.49:0/566425273 reap_dead start 2024-05-15T12:32:50.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.021+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 shutdown_connections 2024-05-15T12:32:50.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.021+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 >> 172.21.15.49:0/566425273 conn(0x7f4d980b4440 msgr2=0x7f4d98069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.026+0000 7ff574ff9640 1 -- 172.21.15.49:0/2673634926 <== osd.3 v2:172.21.15.49:6802/945312262 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff578094e50 con 0x7ff55c0731c0 2024-05-15T12:32:50.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.026+0000 7ff574ff9640 10 client.4318.objecter ms_dispatch 0x7ff578000b90 command_reply(tid 1: 0 ) v1 2024-05-15T12:32:50.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.026+0000 7ff574ff9640 10 client.4318.objecter _finish_command 1 = system:0 2024-05-15T12:32:50.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.026+0000 7ff574ff9640 20 client.4318.objecter put_session s=0x7ff55c072af0 osd=3 4 2024-05-15T12:32:50.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.026+0000 7ff574ff9640 15 client.4318.objecter _session_command_op_remove 3 1 2024-05-15T12:32:50.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 shutdown_connections 2024-05-15T12:32:50.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7f4d9fd9a640 1 -- 172.21.15.49:0/566425273 wait complete. 2024-05-15T12:32:50.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7f4d9fd9a640 1 librados: shutdown 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 10 client.4253.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 15 client.4253.objecter _session_command_op_assign -1 2 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 20 client.4253.objecter _get_session s=0x7fd37c072d40 osd=5 3 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 20 client.4253.objecter put_session s=0x7fd37c072d40 osd=5 3 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 20 client.4253.objecter _get_session s=0x7fd37c072d40 osd=5 3 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 15 client.4253.objecter _session_command_op_remove -1 2 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 20 client.4253.objecter get_session s=0x7fd37c072d40 osd=5 3 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 15 client.4253.objecter _session_command_op_assign 5 2 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 20 client.4253.objecter put_session s=0x7fd37c072d40 osd=5 4 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 10 client.4253.objecter _send_command 2 2024-05-15T12:32:50.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.027+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 --> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd394039d00 con 0x7fd37c073410 2024-05-15T12:32:50.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.029+0000 7fc31dffb640 1 -- 172.21.15.49:0/94651508 <== osd.1 v2:172.21.15.49:6810/3250791401 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc300001240 con 0x7fc30c073290 2024-05-15T12:32:50.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.029+0000 7fc31dffb640 10 client.4278.objecter ms_dispatch 0x7fc328000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:32:50.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.029+0000 7fc31dffb640 10 client.4278.objecter _finish_command 2 = system:0 2024-05-15T12:32:50.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc31dffb640 20 client.4278.objecter put_session s=0x7fc30c072bc0 osd=1 4 2024-05-15T12:32:50.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc31dffb640 15 client.4278.objecter _session_command_op_remove 1 2 2024-05-15T12:32:50.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 10 librados: watch_flush enter 2024-05-15T12:32:50.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 10 librados: watch_flush exit 2024-05-15T12:32:50.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 10 client.4278.objecter close_session for osd.1 2024-05-15T12:32:50.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 1 -- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] conn(0x7fc30c073290 msgr2=0x7fc30c0756b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] conn(0x7fc30c073290 0x7fc30c0756b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 20 client.4278.objecter put_session s=0x7fc30c072bc0 osd=1 1 2024-05-15T12:32:50.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 20 client.4278.objecter shutdown clearing up homeless session... 2024-05-15T12:32:50.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 10 client.4278.objecter successfully canceled tick 2024-05-15T12:32:50.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 1 -- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc30c06ce50 msgr2=0x7fc30c06f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc30c06ce50 0x7fc30c06f310 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc320035000 tx=0x7fc32002e000 comp rx=0 tx=0).stop 2024-05-15T12:32:50.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 10 monclient: shutdown 2024-05-15T12:32:50.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 1 -- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3280af260 msgr2=0x7fc3281d6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc3137fe640 1 --2- 172.21.15.49:0/94651508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3280af260 0x7fc3281d6790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc31800c410 tx=0x7fc31800c8d0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.030+0000 7fc32dada640 1 -- 172.21.15.49:0/94651508 reap_dead start 2024-05-15T12:32:50.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.033+0000 7fc3137fe640 1 -- 172.21.15.49:0/94651508 shutdown_connections 2024-05-15T12:32:50.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.033+0000 7fc3137fe640 1 -- 172.21.15.49:0/94651508 >> 172.21.15.49:0/94651508 conn(0x7fc3280b40d0 msgr2=0x7fc3280b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.033+0000 7fc3137fe640 1 -- 172.21.15.49:0/94651508 shutdown_connections 2024-05-15T12:32:50.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.033+0000 7fc3137fe640 1 -- 172.21.15.49:0/94651508 wait complete. 2024-05-15T12:32:50.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.034+0000 7fc3137fe640 1 librados: shutdown 2024-05-15T12:32:50.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 10 client.4259.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:32:50.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 15 client.4259.objecter _session_command_op_assign -1 2 2024-05-15T12:32:50.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 20 client.4259.objecter _get_session s=0x7f2e1c072b90 osd=0 3 2024-05-15T12:32:50.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 20 client.4259.objecter put_session s=0x7f2e1c072b90 osd=0 3 2024-05-15T12:32:50.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 20 client.4259.objecter _get_session s=0x7f2e1c072b90 osd=0 3 2024-05-15T12:32:50.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 15 client.4259.objecter _session_command_op_remove -1 2 2024-05-15T12:32:50.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 20 client.4259.objecter get_session s=0x7f2e1c072b90 osd=0 3 2024-05-15T12:32:50.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 15 client.4259.objecter _session_command_op_assign 0 2 2024-05-15T12:32:50.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 20 client.4259.objecter put_session s=0x7f2e1c072b90 osd=0 4 2024-05-15T12:32:50.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 10 client.4259.objecter _send_command 2 2024-05-15T12:32:50.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.037+0000 7f2e3cff7640 1 -- 172.21.15.49:0/2760587583 --> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2e300cac10 con 0x7f2e1c073260 2024-05-15T12:32:50.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.038+0000 7fd38affd640 1 -- 172.21.15.49:0/3521073038 <== osd.5 v2:172.21.15.88:6808/3085185190 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd394039d00 con 0x7fd37c073410 2024-05-15T12:32:50.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.038+0000 7fd38affd640 10 client.4253.objecter ms_dispatch 0x7fd39c000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.038+0000 7fd38affd640 10 client.4253.objecter _finish_command 2 = system:0 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.038+0000 7fd38affd640 20 client.4253.objecter put_session s=0x7fd37c072d40 osd=5 4 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.038+0000 7fd38affd640 15 client.4253.objecter _session_command_op_remove 5 2 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.039+0000 7fd3a0e97640 10 librados: watch_flush enter 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.039+0000 7fd3a0e97640 10 librados: watch_flush exit 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.039+0000 7fd3a0e97640 10 client.4253.objecter close_session for osd.5 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.039+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 >> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] conn(0x7fd37c073410 msgr2=0x7fd37c075830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.039+0000 7fd3a0e97640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] conn(0x7fd37c073410 0x7fd37c075830 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.039+0000 7fd3a0e97640 20 client.4253.objecter put_session s=0x7fd37c072d40 osd=5 1 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.039+0000 7fd3a0e97640 20 client.4253.objecter shutdown clearing up homeless session... 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.039+0000 7fd3a0e97640 10 client.4253.objecter successfully canceled tick 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.040+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd37c06cfd0 msgr2=0x7fd37c06f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.040+0000 7fd3a0e97640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd37c06cfd0 0x7fd37c06f490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3940c88e0 tx=0x7fd38c0091c0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.040+0000 7fd3a0e97640 10 monclient: shutdown 2024-05-15T12:32:50.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.040+0000 7fd3a0e97640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.040+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd394007070 msgr2=0x7fd3941609a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.040+0000 7fd3a0e97640 1 --2- 172.21.15.49:0/3521073038 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd394007070 0x7fd3941609a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c410 tx=0x7fd39000c810 comp rx=0 tx=0).stop 2024-05-15T12:32:50.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.040+0000 7fd39ad9d640 1 -- 172.21.15.49:0/3521073038 reap_dead start 2024-05-15T12:32:50.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.041+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 shutdown_connections 2024-05-15T12:32:50.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.041+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 >> 172.21.15.49:0/3521073038 conn(0x7fd394006250 msgr2=0x7fd394032990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.042+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 shutdown_connections 2024-05-15T12:32:50.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.042+0000 7fd3a0e97640 1 -- 172.21.15.49:0/3521073038 wait complete. 2024-05-15T12:32:50.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.043+0000 7fd3a0e97640 1 librados: shutdown 2024-05-15T12:32:50.033 INFO:teuthology.orchestra.run.smithi049.stdout:73014444040 2024-05-15T12:32:50.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-15T12:32:50.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 10 client.4303.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:32:50.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 15 client.4303.objecter _session_command_op_assign -1 2 2024-05-15T12:32:50.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 20 client.4303.objecter _get_session s=0x7f9368076fa0 osd=7 3 2024-05-15T12:32:50.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 20 client.4303.objecter put_session s=0x7f9368076fa0 osd=7 3 2024-05-15T12:32:50.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 20 client.4303.objecter _get_session s=0x7f9368076fa0 osd=7 3 2024-05-15T12:32:50.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 15 client.4303.objecter _session_command_op_remove -1 2 2024-05-15T12:32:50.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 20 client.4303.objecter get_session s=0x7f9368076fa0 osd=7 3 2024-05-15T12:32:50.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 15 client.4303.objecter _session_command_op_assign 7 2 2024-05-15T12:32:50.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 20 client.4303.objecter put_session s=0x7f9368076fa0 osd=7 4 2024-05-15T12:32:50.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 10 client.4303.objecter _send_command 2 2024-05-15T12:32:50.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.048+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 --> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f938804f0e0 con 0x7f9368077670 2024-05-15T12:32:50.041 INFO:teuthology.orchestra.run.smithi049.stdout:90194313225 2024-05-15T12:32:50.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-15T12:32:50.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.056+0000 7f2e23fff640 1 -- 172.21.15.49:0/2760587583 <== osd.0 v2:172.21.15.49:6818/3376151072 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2e300cac10 con 0x7f2e1c073260 2024-05-15T12:32:50.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.056+0000 7f2e23fff640 10 client.4259.objecter ms_dispatch 0x7f2e38000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:32:50.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.056+0000 7f2e23fff640 10 client.4259.objecter _finish_command 2 = system:0 2024-05-15T12:32:50.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.056+0000 7f2e23fff640 20 client.4259.objecter put_session s=0x7f2e1c072b90 osd=0 4 2024-05-15T12:32:50.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.056+0000 7f2e23fff640 15 client.4259.objecter _session_command_op_remove 0 2 2024-05-15T12:32:50.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.059+0000 7f2e21ffb640 10 librados: watch_flush enter 2024-05-15T12:32:50.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-15T12:32:50.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 15 client.4318.objecter _session_command_op_assign -1 2 2024-05-15T12:32:50.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 20 client.4318.objecter _get_session s=0x7ff55c072af0 osd=3 3 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 20 client.4318.objecter put_session s=0x7ff55c072af0 osd=3 3 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 20 client.4318.objecter _get_session s=0x7ff55c072af0 osd=3 3 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 15 client.4318.objecter _session_command_op_remove -1 2 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 20 client.4318.objecter get_session s=0x7ff55c072af0 osd=3 3 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 15 client.4318.objecter _session_command_op_assign 3 2 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 20 client.4318.objecter put_session s=0x7ff55c072af0 osd=3 4 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 10 client.4318.objecter _send_command 2 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.057+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 --> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff5700059c0 con 0x7ff55c0731c0 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.059+0000 7f2e21ffb640 10 librados: watch_flush exit 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.059+0000 7f2e21ffb640 10 client.4259.objecter close_session for osd.0 2024-05-15T12:32:50.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.059+0000 7f2e21ffb640 1 -- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] conn(0x7f2e1c073260 msgr2=0x7f2e1c075680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.059+0000 7f2e21ffb640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] conn(0x7f2e1c073260 0x7f2e1c075680 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.059+0000 7f2e21ffb640 20 client.4259.objecter put_session s=0x7f2e1c072b90 osd=0 1 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.059+0000 7f2e21ffb640 20 client.4259.objecter shutdown clearing up homeless session... 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.059+0000 7f2e21ffb640 10 client.4259.objecter successfully canceled tick 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.059+0000 7f2e21ffb640 1 -- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2e1c06ce20 msgr2=0x7f2e1c06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.059+0000 7f2e21ffb640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2e1c06ce20 0x7f2e1c06f2e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2e24036000 tx=0x7f2e24030000 comp rx=0 tx=0).stop 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.060+0000 7f2e21ffb640 10 monclient: shutdown 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.060+0000 7f2e21ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.060+0000 7f2e21ffb640 1 -- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e300cf5c0 msgr2=0x7f2e300cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.060+0000 7f2e21ffb640 1 --2- 172.21.15.49:0/2760587583 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e300cf5c0 0x7f2e300cb490 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e38066a80 tx=0x7f2e3806a350 comp rx=0 tx=0).stop 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.060+0000 7f2e37fff640 1 -- 172.21.15.49:0/2760587583 reap_dead start 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.060+0000 7f2e21ffb640 1 -- 172.21.15.49:0/2760587583 shutdown_connections 2024-05-15T12:32:50.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.060+0000 7f2e21ffb640 1 -- 172.21.15.49:0/2760587583 >> 172.21.15.49:0/2760587583 conn(0x7f2e300060c0 msgr2=0x7f2e30036e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.061+0000 7f2e21ffb640 1 -- 172.21.15.49:0/2760587583 shutdown_connections 2024-05-15T12:32:50.051 INFO:teuthology.orchestra.run.smithi049.stdout:81604378633 2024-05-15T12:32:50.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-15T12:32:50.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.062+0000 7f2e21ffb640 1 -- 172.21.15.49:0/2760587583 wait complete. 2024-05-15T12:32:50.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.062+0000 7f2e21ffb640 1 librados: shutdown 2024-05-15T12:32:50.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.063+0000 7f937f7fe640 1 -- 172.21.15.49:0/862655462 <== osd.7 v2:172.21.15.88:6824/2037618931 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f938804f0e0 con 0x7f9368077670 2024-05-15T12:32:50.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.063+0000 7f937f7fe640 10 client.4303.objecter ms_dispatch 0x7f9388000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:32:50.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.063+0000 7f937f7fe640 10 client.4303.objecter _finish_command 2 = system:0 2024-05-15T12:32:50.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.063+0000 7f937f7fe640 20 client.4303.objecter put_session s=0x7f9368076fa0 osd=7 4 2024-05-15T12:32:50.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.064+0000 7f937f7fe640 15 client.4303.objecter _session_command_op_remove 7 2 2024-05-15T12:32:50.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 10 librados: watch_flush enter 2024-05-15T12:32:50.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 10 librados: watch_flush exit 2024-05-15T12:32:50.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 10 client.4303.objecter close_session for osd.7 2024-05-15T12:32:50.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 >> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] conn(0x7f9368077670 msgr2=0x7f9368079a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] conn(0x7f9368077670 0x7f9368079a90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 20 client.4303.objecter put_session s=0x7f9368076fa0 osd=7 1 2024-05-15T12:32:50.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 20 client.4303.objecter shutdown clearing up homeless session... 2024-05-15T12:32:50.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 10 client.4303.objecter successfully canceled tick 2024-05-15T12:32:50.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f9368071230 msgr2=0x7f93680736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f9368071230 0x7f93680736f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9380009710 tx=0x7f9380006c90 comp rx=0 tx=0).stop 2024-05-15T12:32:50.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 10 monclient: shutdown 2024-05-15T12:32:50.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9388141b20 msgr2=0x7f93881d70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.065+0000 7f938d1f4640 1 --2- 172.21.15.49:0/862655462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9388141b20 0x7f93881d70d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f937000c390 tx=0x7f937000c850 comp rx=0 tx=0).stop 2024-05-15T12:32:50.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.068+0000 7f938759e640 1 -- 172.21.15.49:0/862655462 reap_dead start 2024-05-15T12:32:50.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.068+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 shutdown_connections 2024-05-15T12:32:50.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.068+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 >> 172.21.15.49:0/862655462 conn(0x7f93880b4440 msgr2=0x7f9388069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.069+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 shutdown_connections 2024-05-15T12:32:50.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.070+0000 7f938d1f4640 1 -- 172.21.15.49:0/862655462 wait complete. 2024-05-15T12:32:50.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.071+0000 7f938d1f4640 1 librados: shutdown 2024-05-15T12:32:50.076 INFO:teuthology.orchestra.run.smithi049.stdout:73014444040 2024-05-15T12:32:50.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-15T12:32:50.081 INFO:teuthology.orchestra.run.smithi049.stdout:81604378634 2024-05-15T12:32:50.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-15T12:32:50.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.102+0000 7ff574ff9640 1 -- 172.21.15.49:0/2673634926 <== osd.3 v2:172.21.15.49:6802/945312262 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff5700059c0 con 0x7ff55c0731c0 2024-05-15T12:32:50.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.102+0000 7ff574ff9640 10 client.4318.objecter ms_dispatch 0x7ff578000b90 command_reply(tid 2: 0 ) v1 2024-05-15T12:32:50.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.102+0000 7ff574ff9640 10 client.4318.objecter _finish_command 2 = system:0 2024-05-15T12:32:50.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.102+0000 7ff574ff9640 20 client.4318.objecter put_session s=0x7ff55c072af0 osd=3 4 2024-05-15T12:32:50.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.102+0000 7ff574ff9640 15 client.4318.objecter _session_command_op_remove 3 2 2024-05-15T12:32:50.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.107+0000 7ff57d531640 10 librados: watch_flush enter 2024-05-15T12:32:50.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.107+0000 7ff57d531640 10 librados: watch_flush exit 2024-05-15T12:32:50.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.107+0000 7ff57d531640 10 client.4318.objecter close_session for osd.3 2024-05-15T12:32:50.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.107+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] conn(0x7ff55c0731c0 msgr2=0x7ff55c0755e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.107+0000 7ff57d531640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] conn(0x7ff55c0731c0 0x7ff55c0755e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.107+0000 7ff57d531640 20 client.4318.objecter put_session s=0x7ff55c072af0 osd=3 1 2024-05-15T12:32:50.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.107+0000 7ff57d531640 20 client.4318.objecter shutdown clearing up homeless session... 2024-05-15T12:32:50.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.107+0000 7ff57d531640 10 client.4318.objecter successfully canceled tick 2024-05-15T12:32:50.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.110+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7ff55c06ce10 msgr2=0x7ff55c06f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.110+0000 7ff57d531640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7ff55c06ce10 0x7ff55c06f2d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff56c036040 tx=0x7ff56c008030 comp rx=0 tx=0).stop 2024-05-15T12:32:50.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.110+0000 7ff57d531640 10 monclient: shutdown 2024-05-15T12:32:50.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.110+0000 7ff57d531640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.110+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5700c1590 msgr2=0x7ff5700c3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.110+0000 7ff57d531640 1 --2- 172.21.15.49:0/2673634926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5700c1590 0x7ff5700c3240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff57806aa70 tx=0x7ff578068a00 comp rx=0 tx=0).stop 2024-05-15T12:32:50.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.110+0000 7ff57cd30640 1 -- 172.21.15.49:0/2673634926 reap_dead start 2024-05-15T12:32:50.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.110+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 shutdown_connections 2024-05-15T12:32:50.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.110+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 >> 172.21.15.49:0/2673634926 conn(0x7ff5700060c0 msgr2=0x7ff5700340f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.112+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 shutdown_connections 2024-05-15T12:32:50.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.112+0000 7ff57d531640 1 -- 172.21.15.49:0/2673634926 wait complete. 2024-05-15T12:32:50.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.112+0000 7ff57d531640 1 librados: shutdown 2024-05-15T12:32:50.120 INFO:teuthology.orchestra.run.smithi049.stdout:64424509445 2024-05-15T12:32:50.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-15T12:32:50.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.205+0000 7f8d8986a640 10 monclient: get_monmap_and_config 2024-05-15T12:32:50.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.205+0000 7f8d8986a640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.205+0000 7f8d8986a640 10 monclient: monmap: 2024-05-15T12:32:50.194 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:50.205961+0000 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:50.205961+0000 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:50.195 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.206+0000 7f8d8986a640 1 Processor -- start 2024-05-15T12:32:50.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.206+0000 7f8d8986a640 1 -- start start 2024-05-15T12:32:50.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.206+0000 7f8d8986a640 10 monclient: init 2024-05-15T12:32:50.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-05-15T12:32:50.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d840b5410 0x7f8d840a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 10 monclient(hunting): picked mon.noname-b con 0x7f8d840b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d8413f990 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:50.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8d84145c00 0x7f8d84146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 10 monclient(hunting): picked mon.noname-c con 0x7f8d84145c00 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:50.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d840ab470 con 0x7f8d8413f990 2024-05-15T12:32:50.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d840b57f0 con 0x7f8d84145c00 2024-05-15T12:32:50.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840a9ea0 con 0x7f8d840b5410 2024-05-15T12:32:50.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.208+0000 7f8d8986a640 10 monclient(hunting): authenticate will time out at 1528.369809s 2024-05-15T12:32:50.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.209+0000 7f8d827fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8d8413f990 0x7f8d8413fdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:36904/0 (socket says 172.21.15.49:36904) 2024-05-15T12:32:50.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.209+0000 7f8d827fc640 1 -- 172.21.15.49:0/177161429 learned_addr learned my addr 172.21.15.49:0/177161429 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:50.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.209+0000 7f8d837fe640 1 --2- 172.21.15.49:0/177161429 >> v2:172.21.15.88:3300/0 conn(0x7f8d84145c00 0x7f8d84146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.210+0000 7f8d837fe640 10 monclient(hunting): get_auth_request con 0x7f8d84145c00 auth_method 0 2024-05-15T12:32:50.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.210+0000 7f8d837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.210+0000 7f8d837fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.210+0000 7f8d837fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.211+0000 7f8d837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.211+0000 7f8d837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.211+0000 7f8d837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.211+0000 7f8d82ffd640 1 --2- 172.21.15.49:0/177161429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d840b5410 0x7f8d840a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.212+0000 7f8d82ffd640 10 monclient(hunting): get_auth_request con 0x7f8d840b5410 auth_method 0 2024-05-15T12:32:50.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.212+0000 7f8d82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.212+0000 7f8d82ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.212+0000 7f8d82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.214+0000 7f8d82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.214+0000 7f8d82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.214+0000 7f8d82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.217+0000 7f8d81ffb640 1 -- 172.21.15.49:0/177161429 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274702655 0 0) 0x7f8d840ab470 con 0x7f8d8413f990 2024-05-15T12:32:50.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.217+0000 7f8d81ffb640 10 monclient(hunting): _negotiate 2024-05-15T12:32:50.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.217+0000 7f8d81ffb640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.217+0000 7f8d81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.217+0000 7f8d81ffb640 10 monclient(hunting): my global_id is 4324 2024-05-15T12:32:50.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.217+0000 7f8d81ffb640 1 -- 172.21.15.49:0/177161429 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d6c003520 con 0x7f8d8413f990 2024-05-15T12:32:50.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.218+0000 7f8d837fe640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-05-15T12:32:50.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.218+0000 7f8d837fe640 1 -- 172.21.15.49:0/177161429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d840b5410 msgr2=0x7f8d840a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.219+0000 7f8d837fe640 1 --2- 172.21.15.49:0/177161429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d840b5410 0x7f8d840a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.219+0000 7f8d837fe640 1 -- 172.21.15.49:0/177161429 >> v1:172.21.15.88:6789/0 conn(0x7f8d8413f990 legacy=0x7f8d8413fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.219+0000 7f8d837fe640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.219+0000 7f8d837fe640 1 monclient: found mon.noname-c 2024-05-15T12:32:50.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.219+0000 7f8d837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.219+0000 7f8d837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.219+0000 7f8d837fe640 1 -- 172.21.15.49:0/177161429 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d840aa0c0 con 0x7f8d84145c00 2024-05-15T12:32:50.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.219+0000 7f8d82ffd640 1 --2- 172.21.15.49:0/177161429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d840b5410 0x7f8d840a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T12:32:50.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.222+0000 7f76e8157640 10 monclient: get_monmap_and_config 2024-05-15T12:32:50.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.222+0000 7f76e8157640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.222+0000 7f76e8157640 10 monclient: monmap: 2024-05-15T12:32:50.211 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.211 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.211 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:50.222824+0000 2024-05-15T12:32:50.212 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:50.222824+0000 2024-05-15T12:32:50.212 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.212 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.212 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:50.212 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:50.212 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:50.212 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:50.212 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:50.212 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.223+0000 7f8d837fe640 1 --2- 172.21.15.49:0/177161429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d84145c00 0x7f8d84146070 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8d780097d0 tx=0x7f8d7802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a0239c10833c74d server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.223+0000 7f76e8157640 1 Processor -- start 2024-05-15T12:32:50.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.224+0000 7f76e8157640 1 -- start start 2024-05-15T12:32:50.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.224+0000 7f76e8157640 10 monclient: init 2024-05-15T12:32:50.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f76d8038db0 0x7f76d80c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 10 monclient(hunting): picked mon.noname-a con 0x7f76d8038db0 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76d8039190 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f76d80c57a0 0x7f76d8034460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 10 monclient(hunting): picked mon.noname-c con 0x7f76d80c57a0 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d80c7ea0 con 0x7f76d8039190 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d80358a0 con 0x7f76d8038db0 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d80360a0 con 0x7f76d80c57a0 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.225+0000 7f76e8157640 10 monclient(hunting): authenticate will time out at 1528.386602s 2024-05-15T12:32:50.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f930c43a640 10 monclient: get_monmap_and_config 2024-05-15T12:32:50.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f930c43a640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f930c43a640 10 monclient: monmap: 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:50.228268+0000 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:50.228268+0000 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:50.218 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d81ffb640 1 -- 172.21.15.49:0/177161429 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78031780 con 0x7f8d84145c00 2024-05-15T12:32:50.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d81ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d81ffb640 10 monclient: dump: 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d81ffb640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d81ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d8986a640 5 monclient: authenticate success, global_id 4321 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d8986a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d81ffb640 1 -- 172.21.15.49:0/177161429 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d78037070 con 0x7f8d84145c00 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d81ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d8986a640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.227+0000 7f8d8986a640 4 set_mon_vals no callback set 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.228+0000 7f8d8986a640 10 monclient: shutdown 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.228+0000 7f8d8986a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.228+0000 7f8d8986a640 1 -- 172.21.15.49:0/177161429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d84145c00 msgr2=0x7f8d84146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f930c43a640 1 Processor -- start 2024-05-15T12:32:50.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.228+0000 7f8d8986a640 1 --2- 172.21.15.49:0/177161429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d84145c00 0x7f8d84146070 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8d780097d0 tx=0x7f8d7802b820 comp rx=0 tx=0).stop 2024-05-15T12:32:50.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f930c43a640 1 -- start start 2024-05-15T12:32:50.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f930c43a640 10 monclient: init 2024-05-15T12:32:50.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f8d8986a640 1 -- 172.21.15.49:0/177161429 shutdown_connections 2024-05-15T12:32:50.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f8d8986a640 1 --2- 172.21.15.49:0/177161429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d84145c00 0x7f8d84146070 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f8d8986a640 1 --2- 172.21.15.49:0/177161429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d840b5410 0x7f8d840a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f8d8986a640 1 -- 172.21.15.49:0/177161429 >> 172.21.15.49:0/177161429 conn(0x7f8d84066f00 msgr2=0x7f8d84067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.228+0000 7f76e66f4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f76d80c57a0 0x7f76d8034460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f76e66f4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f76d80c57a0 0x7f76d8034460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35192/0 (socket says 172.21.15.49:35192) 2024-05-15T12:32:50.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f76e66f4640 1 -- 172.21.15.49:0/4082658602 learned_addr learned my addr 172.21.15.49:0/4082658602 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f76e66f4640 10 monclient(hunting): get_auth_request con 0x7f76d80c57a0 auth_method 0 2024-05-15T12:32:50.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f76e66f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f76e66f4640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.229+0000 7f76e66f4640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f8d8986a640 1 -- 172.21.15.49:0/177161429 shutdown_connections 2024-05-15T12:32:50.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f76e66f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f76e66f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f76e66f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-05-15T12:32:50.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406e830 0x7f930413ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 10 monclient(hunting): picked mon.noname-b con 0x7f930406e830 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f930406ec10 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:50.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f930406fd00 0x7f9304148da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 10 monclient(hunting): picked mon.noname-c con 0x7f930406fd00 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:50.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9304071040 con 0x7f930406ec10 2024-05-15T12:32:50.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93041420c0 con 0x7f930406fd00 2024-05-15T12:32:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93040af070 con 0x7f930406e830 2024-05-15T12:32:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f930c43a640 10 monclient(hunting): authenticate will time out at 1528.392296s 2024-05-15T12:32:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.230+0000 7f8d8986a640 1 -- 172.21.15.49:0/177161429 wait complete. 2024-05-15T12:32:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.231+0000 7f8d8986a640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.231+0000 7f8d8986a640 10 monclient: monmap: 2024-05-15T12:32:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.227 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f8d8986a640 1 librados: starting msgr at 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f76e66f4640 10 monclient(hunting): handle_auth_done global_id 4327 payload 274 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f76e66f4640 1 -- 172.21.15.49:0/4082658602 >> v2:172.21.15.49:3300/0 conn(0x7f76d8038db0 msgr2=0x7f76d80c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f76e66f4640 1 --2- 172.21.15.49:0/4082658602 >> v2:172.21.15.49:3300/0 conn(0x7f76d8038db0 0x7f76d80c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f76e66f4640 1 -- 172.21.15.49:0/4082658602 >> v1:172.21.15.49:6789/0 conn(0x7f76d8039190 legacy=0x7f76d80c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f76e66f4640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f76e66f4640 1 monclient: found mon.noname-c 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f76e66f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f76e66f4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f76e66f4640 1 -- 172.21.15.49:0/4082658602 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d8036220 con 0x7f76d80c57a0 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.233+0000 7f930a9d7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f930406fd00 0x7f9304148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a9d7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f930406fd00 0x7f9304148da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35208/0 (socket says 172.21.15.49:35208) 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a9d7640 1 -- 172.21.15.49:0/2918474061 learned_addr learned my addr 172.21.15.49:0/2918474061 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a9d7640 10 monclient(hunting): get_auth_request con 0x7f930406fd00 auth_method 0 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a9d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a9d7640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a9d7640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a1d6640 1 --2- 172.21.15.49:0/2918474061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406e830 0x7f930413ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f8d8986a640 1 librados: starting objecter 2024-05-15T12:32:50.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.232+0000 7f8d8986a640 1 Processor -- start 2024-05-15T12:32:50.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f8d8986a640 1 -- start start 2024-05-15T12:32:50.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a1d6640 10 monclient(hunting): get_auth_request con 0x7f930406e830 auth_method 0 2024-05-15T12:32:50.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a1d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a1d6640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f930a1d6640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.235+0000 7f930a9d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.235+0000 7f930a9d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.235+0000 7f930a9d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f8d8986a640 1 librados: setting wanted keys 2024-05-15T12:32:50.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f8d8986a640 1 librados: calling monclient init 2024-05-15T12:32:50.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.234+0000 7f8d8986a640 10 monclient: init 2024-05-15T12:32:50.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.235+0000 7f8d8986a640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.235+0000 7f8d8986a640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:32:50.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.236+0000 7f930a1d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.236+0000 7f930a1d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.236+0000 7f930a1d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.235+0000 7f8d8986a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d840b5410 0x7f8d84136830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.235+0000 7f8d8986a640 10 monclient(hunting): picked mon.noname-b con 0x7f8d840b5410 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.235+0000 7f8d8986a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d8413f990 0x7f8d84138f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.236+0000 7f8d8986a640 10 monclient(hunting): picked mon.noname-c con 0x7f8d8413f990 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.237+0000 7f8d8986a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d84145c00 0x7f8d841e46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.237+0000 7f8d8986a640 10 monclient(hunting): picked mon.noname-a con 0x7f8d84145c00 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.237+0000 7f8d8986a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.237+0000 7f8d8986a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840b4aa0 con 0x7f8d84145c00 2024-05-15T12:32:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.237+0000 7f8d8986a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.237+0000 7f8d8986a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840b4920 con 0x7f8d840b5410 2024-05-15T12:32:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.237+0000 7f8d8986a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.237+0000 7f8d8986a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8414cf30 con 0x7f8d8413f990 2024-05-15T12:32:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.237+0000 7f8d8986a640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.237+0000 7f8d8986a640 10 monclient(hunting): authenticate will time out at 1528.398517s 2024-05-15T12:32:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.241+0000 7f930a1d6640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-05-15T12:32:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.241+0000 7f930a1d6640 1 -- 172.21.15.49:0/2918474061 >> v2:172.21.15.88:3300/0 conn(0x7f930406fd00 msgr2=0x7f9304148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.241+0000 7f930a1d6640 1 --2- 172.21.15.49:0/2918474061 >> v2:172.21.15.88:3300/0 conn(0x7f930406fd00 0x7f9304148da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.241+0000 7f930a1d6640 1 -- 172.21.15.49:0/2918474061 >> v1:172.21.15.49:6789/0 conn(0x7f930406ec10 legacy=0x7f93040ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.241+0000 7f930a1d6640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.241+0000 7f930a1d6640 1 monclient: found mon.noname-b 2024-05-15T12:32:50.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.241+0000 7f930a1d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.241+0000 7f930a1d6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.241+0000 7f930a1d6640 1 -- 172.21.15.49:0/2918474061 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930413bfc0 con 0x7f930406e830 2024-05-15T12:32:50.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.244+0000 7f76e66f4640 1 --2- 172.21.15.49:0/4082658602 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d80c57a0 0x7f76d8034460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009870 tx=0x7f76d002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbbbb74e5313fc8b server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.245+0000 7f930a1d6640 1 --2- 172.21.15.49:0/2918474061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406e830 0x7f930413ba80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f92f40097a0 tx=0x7f92f402b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c65b08fdebedef60 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.244+0000 7fb1009d8640 10 monclient: get_monmap_and_config 2024-05-15T12:32:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.244+0000 7fb1009d8640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.244+0000 7fb1009d8640 10 monclient: monmap: 2024-05-15T12:32:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:50.245198+0000 2024-05-15T12:32:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:50.245198+0000 2024-05-15T12:32:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:50.237 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:50.237 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:50.237 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:50.237 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:50.237 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.242+0000 7f8d82ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d840b5410 0x7f8d84136830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.242+0000 7f8d82ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d840b5410 0x7f8d84136830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35220/0 (socket says 172.21.15.49:35220) 2024-05-15T12:32:50.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.242+0000 7f8d82ffd640 1 -- 172.21.15.49:0/801573055 learned_addr learned my addr 172.21.15.49:0/801573055 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.242+0000 7f8d82ffd640 10 monclient(hunting): get_auth_request con 0x7f8d840b5410 auth_method 0 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.242+0000 7f8d82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.242+0000 7f8d82ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.242+0000 7f8d82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.244+0000 7f8d82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.244+0000 7f8d82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.244+0000 7f8d82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.249+0000 7f8d827fc640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d8413f990 0x7f8d84138f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.249+0000 7f8d827fc640 10 monclient(hunting): get_auth_request con 0x7f8d8413f990 auth_method 0 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.249+0000 7f8d827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.249+0000 7f8d827fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.249+0000 7f8d827fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.249+0000 7f8d82ffd640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-05-15T12:32:50.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.249+0000 7f8d82ffd640 1 -- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d8413f990 msgr2=0x7f8d84138f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.249+0000 7f8d82ffd640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d8413f990 0x7f8d84138f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.250+0000 7f8d82ffd640 1 -- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d84145c00 msgr2=0x7f8d841e46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.250+0000 7f8d82ffd640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d84145c00 0x7f8d841e46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.250+0000 7f8d82ffd640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.250+0000 7f8d82ffd640 1 monclient: found mon.noname-b 2024-05-15T12:32:50.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.250+0000 7f8d82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.250+0000 7f8d82ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.250+0000 7f8d82ffd640 1 -- 172.21.15.49:0/801573055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d84137050 con 0x7f8d840b5410 2024-05-15T12:32:50.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.247+0000 7fb1009d8640 1 Processor -- start 2024-05-15T12:32:50.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.247+0000 7fb1009d8640 1 -- start start 2024-05-15T12:32:50.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.247+0000 7fb1009d8640 10 monclient: init 2024-05-15T12:32:50.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0f4034c20 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0f402fca0 0x7fb0f40c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 10 monclient(hunting): picked mon.noname-c con 0x7fb0f402fca0 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0f4032160 0x7fb0f40c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 10 monclient(hunting): picked mon.noname-a con 0x7fb0f4032160 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f40c9c40 con 0x7fb0f4034c20 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0f4035000 con 0x7fb0f4032160 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0f4033240 con 0x7fb0f402fca0 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.248+0000 7fb1009d8640 10 monclient(hunting): authenticate will time out at 1528.410397s 2024-05-15T12:32:50.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e4ef1640 1 -- 172.21.15.49:0/4082658602 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d0035020 con 0x7f76d80c57a0 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e4ef1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e4ef1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e4ef1640 10 monclient: dump: 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.247 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e4ef1640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e4ef1640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e4ef1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e8157640 5 monclient: authenticate success, global_id 4327 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e8157640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e4ef1640 1 -- 172.21.15.49:0/4082658602 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d0031800 con 0x7f76d80c57a0 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e4ef1640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e8157640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.253+0000 7f76e8157640 4 set_mon_vals no callback set 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.254+0000 7f76e8157640 10 monclient: shutdown 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.254+0000 7f76e8157640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.254+0000 7f76e8157640 1 -- 172.21.15.49:0/4082658602 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d80c57a0 msgr2=0x7f76d8034460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.254+0000 7f76e8157640 1 --2- 172.21.15.49:0/4082658602 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d80c57a0 0x7f76d8034460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009870 tx=0x7f76d002b8a0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.255+0000 7f76e8157640 1 -- 172.21.15.49:0/4082658602 shutdown_connections 2024-05-15T12:32:50.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.255+0000 7f76e8157640 1 --2- 172.21.15.49:0/4082658602 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d80c57a0 0x7f76d8034460 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.255+0000 7f76e8157640 1 --2- 172.21.15.49:0/4082658602 >> v2:172.21.15.49:3300/0 conn(0x7f76d8038db0 0x7f76d80c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.255+0000 7f76e8157640 1 -- 172.21.15.49:0/4082658602 >> 172.21.15.49:0/4082658602 conn(0x7f76d8006610 msgr2=0x7f76d8006a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.257+0000 7fb0fbfff640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0f4032160 0x7fb0f40c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.257+0000 7fb0faffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0f402fca0 0x7fb0f40c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.257+0000 7fb0faffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0f402fca0 0x7fb0f40c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35234/0 (socket says 172.21.15.49:35234) 2024-05-15T12:32:50.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.257+0000 7fb0faffd640 1 -- 172.21.15.49:0/1579199157 learned_addr learned my addr 172.21.15.49:0/1579199157 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.257+0000 7fb0faffd640 10 monclient(hunting): get_auth_request con 0x7fb0f402fca0 auth_method 0 2024-05-15T12:32:50.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.257+0000 7fb0faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.257+0000 7fb0faffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.257+0000 7fb0faffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.258+0000 7fb0fbfff640 10 monclient(hunting): get_auth_request con 0x7fb0f4032160 auth_method 0 2024-05-15T12:32:50.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.258+0000 7fb0fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.258+0000 7fb0fbfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.258+0000 7fb0fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f93091d4640 1 -- 172.21.15.49:0/2918474061 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f4038070 con 0x7f930406e830 2024-05-15T12:32:50.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f93091d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f93091d4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:50.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f93091d4640 10 monclient: dump: 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f93091d4640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f93091d4640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f93091d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f930c43a640 5 monclient: authenticate success, global_id 4268 2024-05-15T12:32:50.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f930c43a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:50.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.259+0000 7f8d82ffd640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d840b5410 0x7f8d84136830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009f60 tx=0x7f8d7c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f8d6bfff640 10 client.?.objecter ms_handle_connect 0x7f8d840b5410 2024-05-15T12:32:50.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f8d6bfff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:50.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7f76e8157640 1 -- 172.21.15.49:0/4082658602 shutdown_connections 2024-05-15T12:32:50.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.259+0000 7fb0faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.259+0000 7fb0faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.259+0000 7fb0faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7fb0fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7fb0fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.260+0000 7fb0fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.261+0000 7f93091d4640 1 -- 172.21.15.49:0/2918474061 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f4031710 con 0x7f930406e830 2024-05-15T12:32:50.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.261+0000 7f93091d4640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.261+0000 7f93091d4640 1 -- 172.21.15.49:0/2918474061 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f4031b20 con 0x7f930406e830 2024-05-15T12:32:50.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.262+0000 7f93091d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.262+0000 7f93091d4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:50.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.262+0000 7f93091d4640 10 monclient: dump: 2024-05-15T12:32:50.254 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.254 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.254 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.254 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.255 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.255 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.255 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.255 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.255 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.255 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.262+0000 7f930c43a640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:50.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.262+0000 7f930c43a640 4 set_mon_vals no callback set 2024-05-15T12:32:50.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.261+0000 7fd176629640 10 monclient: get_monmap_and_config 2024-05-15T12:32:50.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.261+0000 7fd176629640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.261+0000 7fd176629640 10 monclient: monmap: 2024-05-15T12:32:50.256 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.256 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.256 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:50.261998+0000 2024-05-15T12:32:50.256 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:50.261998+0000 2024-05-15T12:32:50.256 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.256 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.256 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:50.257 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:50.257 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:50.257 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:50.257 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:50.257 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.263+0000 7fd176629640 1 Processor -- start 2024-05-15T12:32:50.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.263+0000 7f930c43a640 10 monclient: shutdown 2024-05-15T12:32:50.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.263+0000 7f930c43a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.263+0000 7f930c43a640 1 -- 172.21.15.49:0/2918474061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406e830 msgr2=0x7f930413ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.263+0000 7f930c43a640 1 --2- 172.21.15.49:0/2918474061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406e830 0x7f930413ba80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f92f40097a0 tx=0x7f92f402b7b0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f8d6bfff640 1 -- 172.21.15.49:0/801573055 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c018020 con 0x7f8d840b5410 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f8d6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f8d6bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f8d6bfff640 10 monclient: dump: 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f8d6bfff640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f8d6bfff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.263+0000 7f76e8157640 1 -- 172.21.15.49:0/4082658602 wait complete. 2024-05-15T12:32:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f76e8157640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f76e8157640 10 monclient: monmap: 2024-05-15T12:32:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:50.261 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.261 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.261 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:50.261 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:50.261 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:50.261 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f76e8157640 1 librados: starting msgr at 2024-05-15T12:32:50.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f76e8157640 1 librados: starting objecter 2024-05-15T12:32:50.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f76e8157640 1 Processor -- start 2024-05-15T12:32:50.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f8d6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f8d8986a640 5 monclient: authenticate success, global_id 4333 2024-05-15T12:32:50.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f8d8986a640 10 monclient: _renew_subs 2024-05-15T12:32:50.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f8d8986a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f8d8986a640 1 -- 172.21.15.49:0/801573055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d84139bb0 con 0x7f8d840b5410 2024-05-15T12:32:50.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f8d8986a640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:50.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7fb0fa7fc640 1 -- 172.21.15.49:0/1579199157 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020138578 0 0) 0x7fb0f40c9c40 con 0x7fb0f4034c20 2024-05-15T12:32:50.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7fb0fa7fc640 10 monclient(hunting): _negotiate 2024-05-15T12:32:50.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7fb0fa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7fb0fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7fb0fa7fc640 10 monclient(hunting): my global_id is 4339 2024-05-15T12:32:50.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7fb0fa7fc640 1 -- 172.21.15.49:0/1579199157 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ec003520 con 0x7fb0f4034c20 2024-05-15T12:32:50.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f8d8986a640 10 monclient: _renew_subs 2024-05-15T12:32:50.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f8d8986a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f8d8986a640 1 -- 172.21.15.49:0/801573055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d84137aa0 con 0x7f8d840b5410 2024-05-15T12:32:50.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f8d8986a640 1 librados: init done 2024-05-15T12:32:50.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f76e8157640 1 -- start start 2024-05-15T12:32:50.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f76e8157640 1 librados: setting wanted keys 2024-05-15T12:32:50.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f76e8157640 1 librados: calling monclient init 2024-05-15T12:32:50.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.264+0000 7f76e8157640 10 monclient: init 2024-05-15T12:32:50.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d8038db0 0x7f76d8009e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 10 monclient(hunting): picked mon.noname-b con 0x7f76d8038db0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d800c860 0x7f76d800ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 10 monclient(hunting): picked mon.noname-c con 0x7f76d800c860 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d800b050 0x7f76d816a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 10 monclient(hunting): picked mon.noname-a con 0x7f76d800b050 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80364c0 con 0x7f76d800b050 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80c8cb0 con 0x7f76d8038db0 2024-05-15T12:32:50.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d8036640 con 0x7f76d800c860 2024-05-15T12:32:50.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.265+0000 7f76e8157640 10 monclient(hunting): authenticate will time out at 1528.427205s 2024-05-15T12:32:50.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f930c43a640 1 -- 172.21.15.49:0/2918474061 shutdown_connections 2024-05-15T12:32:50.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.263+0000 7fd176629640 1 -- start start 2024-05-15T12:32:50.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.263+0000 7fd176629640 10 monclient: init 2024-05-15T12:32:50.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-05-15T12:32:50.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd170071d00 0x7fd17013b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 10 monclient(hunting): picked mon.noname-c con 0x7fd170071d00 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:50.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd17013ba30 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:50.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 0x7fd170148f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 10 monclient(hunting): picked mon.noname-b con 0x7fd17013ed00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd170070fb0 con 0x7fd17013ba30 2024-05-15T12:32:50.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd170071290 con 0x7fd170071d00 2024-05-15T12:32:50.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd170071570 con 0x7fd17013ed00 2024-05-15T12:32:50.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fd176629640 10 monclient(hunting): authenticate will time out at 1528.427659s 2024-05-15T12:32:50.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f930c43a640 1 --2- 172.21.15.49:0/2918474061 >> v2:172.21.15.88:3300/0 conn(0x7f930406fd00 0x7f9304148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f930c43a640 1 --2- 172.21.15.49:0/2918474061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406e830 0x7f930413ba80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f930c43a640 1 -- 172.21.15.49:0/2918474061 >> 172.21.15.49:0/2918474061 conn(0x7f93040b3ca0 msgr2=0x7f93040b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f930c43a640 1 -- 172.21.15.49:0/2918474061 shutdown_connections 2024-05-15T12:32:50.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f8d6bfff640 1 -- 172.21.15.49:0/801573055 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c014680 con 0x7f8d840b5410 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f8d6bfff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f8d6bfff640 1 -- 172.21.15.49:0/801573055 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c014a70 con 0x7f8d840b5410 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f8d6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f8d6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7f8d6bfff640 10 monclient: dump: 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.270 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.267+0000 7f8d817fa640 4 set_mon_vals no callback set 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fb0faffd640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fb0faffd640 1 -- 172.21.15.49:0/1579199157 >> v2:172.21.15.49:3300/0 conn(0x7fb0f4032160 msgr2=0x7fb0f40c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fb0faffd640 1 --2- 172.21.15.49:0/1579199157 >> v2:172.21.15.49:3300/0 conn(0x7fb0f4032160 0x7fb0f40c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fb0faffd640 1 -- 172.21.15.49:0/1579199157 >> v1:172.21.15.88:6789/0 conn(0x7fb0f4034c20 legacy=0x7fb0f40cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fb0faffd640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fb0faffd640 1 monclient: found mon.noname-c 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fb0faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fb0faffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.266+0000 7fb0faffd640 1 -- 172.21.15.49:0/1579199157 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4031c40 con 0x7fb0f402fca0 2024-05-15T12:32:50.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7fd174bc6640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 0x7fd170148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7fd174bc6640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 0x7fd170148f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46130/0 (socket says 172.21.15.49:46130) 2024-05-15T12:32:50.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7fd174bc6640 1 -- 172.21.15.49:0/369259908 learned_addr learned my addr 172.21.15.49:0/369259908 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7fd174bc6640 10 monclient(hunting): get_auth_request con 0x7fd17013ed00 auth_method 0 2024-05-15T12:32:50.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7fd174bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7fd174bc6640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7fd174bc6640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.268+0000 7f930c43a640 1 -- 172.21.15.49:0/2918474061 wait complete. 2024-05-15T12:32:50.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f76e5ef3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d8038db0 0x7f76d8009e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f76e56f2640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d800c860 0x7f76d800ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f76e56f2640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d800c860 0x7f76d800ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46114/0 (socket says 172.21.15.49:46114) 2024-05-15T12:32:50.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f76e56f2640 1 -- 172.21.15.49:0/4005751253 learned_addr learned my addr 172.21.15.49:0/4005751253 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f76e56f2640 10 monclient(hunting): get_auth_request con 0x7f76d800c860 auth_method 0 2024-05-15T12:32:50.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f76e56f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f76e56f2640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f76e56f2640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f930c43a640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f930c43a640 10 monclient: monmap: 2024-05-15T12:32:50.274 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.274 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:50.274 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:50.275 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:50.275 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.275 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.275 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:50.275 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:50.275 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:50.275 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f930c43a640 1 librados: starting msgr at 2024-05-15T12:32:50.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f930c43a640 1 librados: starting objecter 2024-05-15T12:32:50.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.269+0000 7f930c43a640 1 Processor -- start 2024-05-15T12:32:50.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7f8d69ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7f8d69ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7f8d69ffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7f8d69ffb640 1 -- 172.21.15.49:0/801573055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8404f0e0 con 0x7f8d840b5410 2024-05-15T12:32:50.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7f930c43a640 1 -- start start 2024-05-15T12:32:50.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7f930c43a640 1 librados: setting wanted keys 2024-05-15T12:32:50.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7f930c43a640 1 librados: calling monclient init 2024-05-15T12:32:50.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7f930c43a640 10 monclient: init 2024-05-15T12:32:50.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7fd174bc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7fd174bc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.270+0000 7fd174bc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f76e56f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f76e56f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f76e56f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:32:50.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f930406e830 0x7f9304134510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 10 monclient(hunting): picked mon.noname-b con 0x7f930406e830 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:50.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f930406ec10 0x7f93041415b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 10 monclient(hunting): picked mon.noname-a con 0x7f930406ec10 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:50.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.272+0000 7f76e5ef3640 10 monclient(hunting): get_auth_request con 0x7f76d8038db0 auth_method 0 2024-05-15T12:32:50.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.272+0000 7f76e5ef3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.272+0000 7f76e5ef3640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.272+0000 7f76e5ef3640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406fd00 0x7f93041d78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 10 monclient(hunting): picked mon.noname-c con 0x7f930406fd00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93040b47c0 con 0x7f930406ec10 2024-05-15T12:32:50.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930406f280 con 0x7f930406e830 2024-05-15T12:32:50.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93040b4940 con 0x7f930406fd00 2024-05-15T12:32:50.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.272+0000 7f76e5ef3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.272+0000 7f76e5ef3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.272+0000 7f76e5ef3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.271+0000 7f930c43a640 10 monclient(hunting): authenticate will time out at 1528.433212s 2024-05-15T12:32:50.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f82432cf640 10 monclient: get_monmap_and_config 2024-05-15T12:32:50.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f82432cf640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f930a9d7640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406fd00 0x7f93041d78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f82432cf640 10 monclient: monmap: 2024-05-15T12:32:50.283 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.283 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.283 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:50.277696+0000 2024-05-15T12:32:50.283 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:50.277696+0000 2024-05-15T12:32:50.283 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.283 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.283 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:50.284 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:50.284 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:50.284 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:50.284 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:50.284 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f930a9d7640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406fd00 0x7f93041d78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46136/0 (socket says 172.21.15.49:46136) 2024-05-15T12:32:50.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f930a9d7640 1 -- 172.21.15.49:0/4059849635 learned_addr learned my addr 172.21.15.49:0/4059849635 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f76e66f4640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d800b050 0x7f76d816a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f930a9d7640 10 monclient(hunting): get_auth_request con 0x7f930406fd00 auth_method 0 2024-05-15T12:32:50.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f930a9d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f930a9d7640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f930a9d7640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f76e66f4640 10 monclient(hunting): get_auth_request con 0x7f76d800b050 auth_method 0 2024-05-15T12:32:50.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f76e66f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f76e66f4640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7f76e66f4640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7fd174bc6640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-05-15T12:32:50.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.277+0000 7fd174bc6640 1 -- 172.21.15.49:0/369259908 >> v2:172.21.15.88:3300/0 conn(0x7fd170071d00 msgr2=0x7fd17013b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7fd174bc6640 1 --2- 172.21.15.49:0/369259908 >> v2:172.21.15.88:3300/0 conn(0x7fd170071d00 0x7fd17013b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7fd174bc6640 1 -- 172.21.15.49:0/369259908 >> v1:172.21.15.88:6789/0 conn(0x7fd17013ba30 legacy=0x7fd17013e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7fd174bc6640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7fd174bc6640 1 monclient: found mon.noname-b 2024-05-15T12:32:50.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7fd174bc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7fd174bc6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7fd174bc6640 1 -- 172.21.15.49:0/369259908 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd170072490 con 0x7fd17013ed00 2024-05-15T12:32:50.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7f82432cf640 1 Processor -- start 2024-05-15T12:32:50.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7f930a9d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7f930a9d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e66f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e66f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e66f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.278+0000 7f930a9d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f93099d5640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f930406ec10 0x7f93041415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f82432cf640 1 -- start start 2024-05-15T12:32:50.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f82432cf640 10 monclient: init 2024-05-15T12:32:50.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e5ef3640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-05-15T12:32:50.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e5ef3640 1 -- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d800c860 msgr2=0x7f76d800ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e5ef3640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d800c860 0x7f76d800ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e5ef3640 1 -- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d800b050 msgr2=0x7f76d816a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e5ef3640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d800b050 0x7f76d816a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e5ef3640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e5ef3640 1 monclient: found mon.noname-b 2024-05-15T12:32:50.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e5ef3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e5ef3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.279+0000 7f76e5ef3640 1 -- 172.21.15.49:0/4005751253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d800c440 con 0x7f76d8038db0 2024-05-15T12:32:50.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f930a1d6640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f930406e830 0x7f9304134510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-05-15T12:32:50.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f823c0b53a0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:50.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c1438d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 10 monclient(hunting): picked mon.noname-b con 0x7f823c147210 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f823c143e10 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:50.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823c06de70 con 0x7f823c0b53a0 2024-05-15T12:32:50.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823c0b4820 con 0x7f823c143e10 2024-05-15T12:32:50.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823c06e0e0 con 0x7f823c147210 2024-05-15T12:32:50.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.280+0000 7f82432cf640 10 monclient(hunting): authenticate will time out at 1528.442016s 2024-05-15T12:32:50.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.281+0000 7f930a1d6640 10 monclient(hunting): get_auth_request con 0x7f930406e830 auth_method 0 2024-05-15T12:32:50.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.281+0000 7f930a1d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.281+0000 7f930a1d6640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.281+0000 7f930a1d6640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.283+0000 7f930a9d7640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-05-15T12:32:50.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.283+0000 7f930a9d7640 1 -- 172.21.15.49:0/4059849635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f930406e830 msgr2=0x7f9304134510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.283+0000 7f930a9d7640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f930406e830 0x7f9304134510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.283+0000 7f930a9d7640 1 -- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f930406ec10 msgr2=0x7f93041415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.283+0000 7f930a9d7640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f930406ec10 0x7f93041415b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.283+0000 7f930a9d7640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.283+0000 7f930a9d7640 1 monclient: found mon.noname-c 2024-05-15T12:32:50.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.283+0000 7f930a9d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.283+0000 7f930a9d7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.283+0000 7f930a9d7640 1 -- 172.21.15.49:0/4059849635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93041d7f10 con 0x7f930406fd00 2024-05-15T12:32:50.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.286+0000 7fd174bc6640 1 --2- 172.21.15.49:0/369259908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 0x7fd170148f00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd1680091e0 tx=0x7fd16802cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3c20bacc213ac96 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.286+0000 7f55f18cf640 10 monclient: get_monmap_and_config 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.286+0000 7f55f18cf640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.286+0000 7f55f18cf640 10 monclient: monmap: 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:50.287463+0000 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:50.287463+0000 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:50.295 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:50.296 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:50.296 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:50.296 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:50.296 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.287+0000 7f930a9d7640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406fd00 0x7f93041d78f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00b080 tx=0x7f92fc00b540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.287+0000 7f92faffd640 10 client.?.objecter ms_handle_connect 0x7f930406fd00 2024-05-15T12:32:50.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.287+0000 7f92faffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:50.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.287+0000 7f55f18cf640 1 Processor -- start 2024-05-15T12:32:50.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.287+0000 7f55f18cf640 1 -- start start 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.287+0000 7f55f18cf640 10 monclient: init 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec0b5410 0x7f55ec0a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 10 monclient(hunting): picked mon.noname-b con 0x7f55ec0b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55ec13f990 0x7f55ec13fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 10 monclient(hunting): picked mon.noname-c con 0x7f55ec13f990 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55ec145c00 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ec140310 con 0x7f55ec145c00 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ec0b57f0 con 0x7f55ec13f990 2024-05-15T12:32:50.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0a9ea0 con 0x7f55ec0b5410 2024-05-15T12:32:50.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.288+0000 7f55f18cf640 10 monclient(hunting): authenticate will time out at 1528.450247s 2024-05-15T12:32:50.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.289+0000 7f824106b640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f823c0b53a0 0x7f823c146b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34630/0 (socket says 172.21.15.49:34630) 2024-05-15T12:32:50.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.289+0000 7f824106b640 1 -- 172.21.15.49:0/2453766991 learned_addr learned my addr 172.21.15.49:0/2453766991 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f92faffd640 1 -- 172.21.15.49:0/4059849635 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc01b070 con 0x7f930406fd00 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f92faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f92faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f92faffd640 10 monclient: dump: 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.300 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f92faffd640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f92faffd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f92faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f930c43a640 5 monclient: authenticate success, global_id 4280 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f930c43a640 10 monclient: _renew_subs 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f930c43a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f930c43a640 1 -- 172.21.15.49:0/4059849635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93041d8490 con 0x7f930406fd00 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f930c43a640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f930c43a640 10 monclient: _renew_subs 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f930c43a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f930c43a640 1 -- 172.21.15.49:0/4059849635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9304132430 con 0x7f930406fd00 2024-05-15T12:32:50.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.297+0000 7f930c43a640 1 librados: init done 2024-05-15T12:32:50.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.298+0000 7f930c43a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.298+0000 7f930c43a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.298+0000 7f930c43a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.299+0000 7f930c43a640 1 -- 172.21.15.49:0/4059849635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930404f0e0 con 0x7f930406fd00 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.299+0000 7f824086a640 1 --2- 172.21.15.49:0/2453766991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c1438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.300+0000 7f824086a640 10 monclient(hunting): get_auth_request con 0x7f823c147210 auth_method 0 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.300+0000 7f824086a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.300+0000 7f824086a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.300+0000 7f824086a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.301+0000 7f824086a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.301+0000 7f824086a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.301+0000 7f824086a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.305+0000 7f824086a640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.305+0000 7f824086a640 1 -- 172.21.15.49:0/2453766991 >> v1:172.21.15.88:6789/0 conn(0x7f823c143e10 legacy=0x7f823c1515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.305+0000 7f824086a640 1 -- 172.21.15.49:0/2453766991 >> v1:172.21.15.49:6789/0 conn(0x7f823c0b53a0 legacy=0x7f823c146b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.305+0000 7f824086a640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.305+0000 7f824086a640 1 monclient: found mon.noname-b 2024-05-15T12:32:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.305+0000 7f824086a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.305+0000 7f824086a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.305+0000 7f824086a640 1 -- 172.21.15.49:0/2453766991 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c0b4300 con 0x7f823c147210 2024-05-15T12:32:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.306+0000 7f92faffd640 1 -- 172.21.15.49:0/4059849635 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc003da0 con 0x7f930406fd00 2024-05-15T12:32:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.306+0000 7f92faffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.306+0000 7f93089d3640 4 set_mon_vals no callback set 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.308+0000 7f92faffd640 1 -- 172.21.15.49:0/4059849635 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc025060 con 0x7f930406fd00 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.308+0000 7f92faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.308+0000 7f92faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.308+0000 7f92faffd640 10 monclient: dump: 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.305 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.309+0000 7fd16effd640 1 -- 172.21.15.49:0/369259908 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16803a070 con 0x7fd17013ed00 2024-05-15T12:32:50.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.309+0000 7fd16effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.309+0000 7fd16effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:50.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.309+0000 7fd16effd640 10 monclient: dump: 2024-05-15T12:32:50.306 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.306 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.306 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.306 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.309+0000 7fd16effd640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.309+0000 7fd16effd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.309+0000 7fd16effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.310+0000 7fd176629640 5 monclient: authenticate success, global_id 4274 2024-05-15T12:32:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.310+0000 7fd176629640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.310+0000 7f824086a640 1 --2- 172.21.15.49:0/2453766991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c1438d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f82380097d0 tx=0x7f823802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=427e4c21d97870fc server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.311+0000 7f55eb7fe640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f55ec145c00 0x7f55ec146030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34644/0 (socket says 172.21.15.49:34644) 2024-05-15T12:32:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.312+0000 7f55eb7fe640 1 -- 172.21.15.49:0/2881221431 learned_addr learned my addr 172.21.15.49:0/2881221431 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.315+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935447877 0 0) 0x7f55ec140310 con 0x7f55ec145c00 2024-05-15T12:32:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.315+0000 7f55e9ffb640 10 monclient(hunting): _negotiate 2024-05-15T12:32:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.315+0000 7f55e9ffb640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.315+0000 7f55e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.315+0000 7f55e9ffb640 10 monclient(hunting): my global_id is 4296 2024-05-15T12:32:50.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.315+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55d4003520 con 0x7f55ec145c00 2024-05-15T12:32:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 <== mon.0 v1:172.21.15.49:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2138203916 0 0) 0x7f55d4003520 con 0x7f55ec145c00 2024-05-15T12:32:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec0b5410 msgr2=0x7f55ec0a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 1 --2- 172.21.15.49:0/2881221431 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec0b5410 0x7f55ec0a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 >> v2:172.21.15.88:3300/0 conn(0x7f55ec13f990 msgr2=0x7f55ec13fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 1 --2- 172.21.15.49:0/2881221431 >> v2:172.21.15.88:3300/0 conn(0x7f55ec13f990 0x7f55ec13fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 1 monclient: found mon.noname-a-legacy 2024-05-15T12:32:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.49:6789/0 2024-05-15T12:32:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 --> v1:172.21.15.49:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec0ab470 con 0x7f55ec145c00 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55f18cf640 5 monclient: authenticate success, global_id 4296 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55f18cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.320+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 <== mon.0 v1:172.21.15.49:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2686122088 0 0) 0x7f55e40019f0 con 0x7f55ec145c00 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient: dump: 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.311 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] but i'm connected to v1:172.21.15.49:6789/0, reconnecting 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 >> v1:172.21.15.49:6789/0 conn(0x7f55ec145c00 legacy=0x7f55ec146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 1 --2- 172.21.15.49:0/2881221431 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec0b5410 0x7f55d4003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient(hunting): picked mon.c con 0x7f55ec0b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 1 --2- 172.21.15.49:0/2881221431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 0x7f55d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient(hunting): picked mon.a con 0x7f55d4004170 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 1 --2- 172.21.15.49:0/2881221431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55d4004e60 0x7f55d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient(hunting): picked mon.b con 0x7f55d4004e60 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:50.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0b57f0 con 0x7f55d4004170 2024-05-15T12:32:50.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0a9ea0 con 0x7f55d4004e60 2024-05-15T12:32:50.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d400bf00 con 0x7f55ec0b5410 2024-05-15T12:32:50.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55e9ffb640 10 monclient(hunting): _finish_auth 0 2024-05-15T12:32:50.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55f18cf640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:50.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.321+0000 7f55ea7fc640 1 --2- 172.21.15.49:0/2881221431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 0x7f55d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.322+0000 7f55ea7fc640 10 monclient(hunting): get_auth_request con 0x7f55d4004170 auth_method 0 2024-05-15T12:32:50.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.322+0000 7f55ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.322+0000 7f55ea7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.322+0000 7f55ea7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:32:50.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.324+0000 7f55ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.324+0000 7f55ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.324+0000 7f55ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:32:50.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.325+0000 7f55ea7fc640 10 monclient(hunting): handle_auth_done global_id 4296 payload 293 2024-05-15T12:32:50.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.325+0000 7f55ea7fc640 1 -- 172.21.15.49:0/2881221431 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec0b5410 msgr2=0x7f55d4003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 1 --2- 172.21.15.49:0/2881221431 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec0b5410 0x7f55d4003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 1 -- 172.21.15.49:0/2881221431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55d4004e60 msgr2=0x7f55d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 1 --2- 172.21.15.49:0/2881221431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55d4004e60 0x7f55d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 1 monclient: found mon.a 2024-05-15T12:32:50.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 1 -- 172.21.15.49:0/2881221431 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec0ab470 con 0x7f55d4004170 2024-05-15T12:32:50.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.326+0000 7f55ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.328+0000 7f55ea7fc640 1 --2- 172.21.15.49:0/2881221431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 0x7f55d4004800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009860 tx=0x7f55d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7466063ddad8008 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.331+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc037070 con 0x7f55d4004170 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.331+0000 7f55e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.331+0000 7f55e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.331+0000 7f55e9ffb640 10 monclient: dump: 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.320 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.321 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.321 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.331+0000 7f55f18cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:50.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.332+0000 7f4eb32d6640 10 monclient: get_monmap_and_config 2024-05-15T12:32:50.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.332+0000 7f4eb32d6640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.332+0000 7f4eb32d6640 10 monclient: monmap: 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:50.332737+0000 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:50.332737+0000 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:50.322 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.332+0000 7f92faffd640 1 -- 172.21.15.49:0/4059849635 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f92fc005d10 con 0x7f930406fd00 2024-05-15T12:32:50.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.332+0000 7f92faffd640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f92e806cdd0 0x7f92e806f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.333+0000 7f930a1d6640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f92e806cdd0 0x7f92e806f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.333+0000 7f930a1d6640 10 monclient: get_auth_request con 0x7f92e806cdd0 auth_method 0 2024-05-15T12:32:50.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.333+0000 7f4eb32d6640 1 Processor -- start 2024-05-15T12:32:50.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.333+0000 7f4eb32d6640 1 -- start start 2024-05-15T12:32:50.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.333+0000 7f4eb32d6640 10 monclient: init 2024-05-15T12:32:50.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.333+0000 7f92faffd640 10 client.4280.objecter ms_handle_connect 0x7f92e806cdd0 2024-05-15T12:32:50.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.333+0000 7f930a1d6640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f92e806cdd0 0x7f92e806f290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f92f4035000 tx=0x7f92f4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55dc032600 con 0x7f55d4004170 2024-05-15T12:32:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f55e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f55e9ffb640 1 -- 172.21.15.49:0/2881221431 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc037070 con 0x7f55d4004170 2024-05-15T12:32:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f55e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f55e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f55e9ffb640 10 monclient: dump: 2024-05-15T12:32:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.326 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.326 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f55f18cf640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:50.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f55f18cf640 4 set_mon_vals no callback set 2024-05-15T12:32:50.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4eac072210 0x7f4eac06fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 10 monclient(hunting): picked mon.noname-a con 0x7f4eac072210 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4eac1420e0 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4eac074560 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac071a80 con 0x7f4eac074560 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac070380 con 0x7f4eac1420e0 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac06f2e0 con 0x7f4eac072210 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.334+0000 7f4eb32d6640 10 monclient(hunting): authenticate will time out at 1528.495826s 2024-05-15T12:32:50.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.335+0000 7f55f18cf640 10 monclient: shutdown 2024-05-15T12:32:50.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.335+0000 7f55f18cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.335+0000 7f55f18cf640 1 -- 172.21.15.49:0/2881221431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 msgr2=0x7f55d4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.335+0000 7f55f18cf640 1 --2- 172.21.15.49:0/2881221431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 0x7f55d4004800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009860 tx=0x7f55d4005830 comp rx=0 tx=0).stop 2024-05-15T12:32:50.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.335+0000 7f55eb7fe640 1 -- 172.21.15.49:0/2881221431 reap_dead start 2024-05-15T12:32:50.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.335+0000 7f55f18cf640 1 -- 172.21.15.49:0/2881221431 shutdown_connections 2024-05-15T12:32:50.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.335+0000 7f55f18cf640 1 -- 172.21.15.49:0/2881221431 >> 172.21.15.49:0/2881221431 conn(0x7f55ec066f00 msgr2=0x7f55ec067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.335+0000 7f55f18cf640 1 -- 172.21.15.49:0/2881221431 shutdown_connections 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.335+0000 7f55f18cf640 1 -- 172.21.15.49:0/2881221431 wait complete. 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.336+0000 7f55f18cf640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.336+0000 7f55f18cf640 10 monclient: monmap: 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.336+0000 7f55f18cf640 1 librados: starting msgr at 2024-05-15T12:32:50.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.336+0000 7f55f18cf640 1 librados: starting objecter 2024-05-15T12:32:50.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.336+0000 7f55f18cf640 1 Processor -- start 2024-05-15T12:32:50.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7fd16effd640 1 -- 172.21.15.49:0/369259908 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16802d950 con 0x7fd17013ed00 2024-05-15T12:32:50.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7fd16effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7fd176629640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:50.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7fd176629640 4 set_mon_vals no callback set 2024-05-15T12:32:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7f55f18cf640 1 -- start start 2024-05-15T12:32:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7f55f18cf640 1 librados: setting wanted keys 2024-05-15T12:32:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7f55f18cf640 1 librados: calling monclient init 2024-05-15T12:32:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7f55f18cf640 10 monclient: init 2024-05-15T12:32:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7f4eb1873640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f4eac074560 0x7f4eac0ae4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34650/0 (socket says 172.21.15.49:34650) 2024-05-15T12:32:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7f4eb1873640 1 -- 172.21.15.49:0/161625110 learned_addr learned my addr 172.21.15.49:0/161625110 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7fd176629640 10 monclient: shutdown 2024-05-15T12:32:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7fd176629640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7fd176629640 1 -- 172.21.15.49:0/369259908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 msgr2=0x7fd170148f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.337+0000 7fd176629640 1 --2- 172.21.15.49:0/369259908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 0x7fd170148f00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd1680091e0 tx=0x7fd16802cba0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7fd176629640 1 -- 172.21.15.49:0/369259908 shutdown_connections 2024-05-15T12:32:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7fd176629640 1 --2- 172.21.15.49:0/369259908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 0x7fd170148f00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7fd176629640 1 --2- 172.21.15.49:0/369259908 >> v2:172.21.15.88:3300/0 conn(0x7fd170071d00 0x7fd17013b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7fd176629640 1 -- 172.21.15.49:0/369259908 >> 172.21.15.49:0/369259908 conn(0x7fd1700b3e10 msgr2=0x7fd1700b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7fd176629640 1 -- 172.21.15.49:0/369259908 shutdown_connections 2024-05-15T12:32:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7fd176629640 1 -- 172.21.15.49:0/369259908 wait complete. 2024-05-15T12:32:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:32:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f4eb1072640 1 --2- 172.21.15.49:0/161625110 >> v2:172.21.15.49:3300/0 conn(0x7f4eac072210 0x7f4eac06fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 0x7f55ec142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 10 monclient(hunting): picked mon.noname-a con 0x7f55d4004170 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec140500 0x7f55ec0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 10 monclient(hunting): picked mon.noname-c con 0x7f55ec140500 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ec0b87e0 0x7f55ec1d94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 10 monclient(hunting): picked mon.noname-b con 0x7f55ec0b87e0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0b4920 con 0x7f55d4004170 2024-05-15T12:32:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0aa620 con 0x7f55ec0b87e0 2024-05-15T12:32:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0b4aa0 con 0x7f55ec140500 2024-05-15T12:32:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.338+0000 7f55f18cf640 10 monclient(hunting): authenticate will time out at 1528.500063s 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7fd176629640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7fd176629640 10 monclient: monmap: 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:50.338 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7f4eb1072640 10 monclient(hunting): get_auth_request con 0x7f4eac072210 auth_method 0 2024-05-15T12:32:50.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7f4eb1072640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7f4eb1072640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7f4eb1072640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7fd176629640 1 librados: starting msgr at 2024-05-15T12:32:50.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7fd176629640 1 librados: starting objecter 2024-05-15T12:32:50.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7fd176629640 1 Processor -- start 2024-05-15T12:32:50.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7f4eb1072640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7f4eb1072640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7f4eb1072640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7fd176629640 1 -- start start 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7fd176629640 1 librados: setting wanted keys 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7fd176629640 1 librados: calling monclient init 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.339+0000 7fd176629640 10 monclient: init 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd170071d00 0x7fd1700b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 10 monclient(hunting): picked mon.noname-a con 0x7fd170071d00 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd17013ba30 0x7fd1700b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 10 monclient(hunting): picked mon.noname-b con 0x7fd17013ba30 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:50.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 0x7fd17012e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 10 monclient(hunting): picked mon.noname-c con 0x7fd17013ed00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17013f300 con 0x7fd170071d00 2024-05-15T12:32:50.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17013f180 con 0x7fd17013ba30 2024-05-15T12:32:50.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700720e0 con 0x7fd17013ed00 2024-05-15T12:32:50.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.340+0000 7fd176629640 10 monclient(hunting): authenticate will time out at 1528.502019s 2024-05-15T12:32:50.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.342+0000 7f4eb1072640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-05-15T12:32:50.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.342+0000 7f4eb1072640 1 -- 172.21.15.49:0/161625110 >> v1:172.21.15.88:6789/0 conn(0x7f4eac1420e0 legacy=0x7f4eac0725f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:50.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.342+0000 7f4eb1072640 1 -- 172.21.15.49:0/161625110 >> v1:172.21.15.49:6789/0 conn(0x7f4eac074560 legacy=0x7f4eac0ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.343+0000 7f4eb1072640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.343+0000 7f4eb1072640 1 monclient: found mon.noname-a 2024-05-15T12:32:50.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.343+0000 7f4eb1072640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.343+0000 7f4eb1072640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.343+0000 7f4eb1072640 1 -- 172.21.15.49:0/161625110 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac142550 con 0x7f4eac072210 2024-05-15T12:32:50.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.347+0000 7f4eb1072640 1 --2- 172.21.15.49:0/161625110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac072210 0x7f4eac06fae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4ea80097a0 tx=0x7f4ea802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec32879421ffb132 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.352+0000 7f55eaffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 0x7f55ec142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.352+0000 7f55eaffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 0x7f55ec142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39062/0 (socket says 172.21.15.49:39062) 2024-05-15T12:32:50.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.352+0000 7f55eaffd640 1 -- 172.21.15.49:0/900168732 learned_addr learned my addr 172.21.15.49:0/900168732 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.352+0000 7fd16ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd170071d00 0x7fd1700b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.352+0000 7f55eaffd640 10 monclient(hunting): get_auth_request con 0x7f55d4004170 auth_method 0 2024-05-15T12:32:50.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.352+0000 7f55eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.352+0000 7f55eaffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.352+0000 7f55eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 1 -- 172.21.15.49:0/161625110 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8035020 con 0x7f4eac072210 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 10 monclient: dump: 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.347 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 1 -- 172.21.15.49:0/161625110 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea8031830 con 0x7f4eac072210 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 1 -- 172.21.15.49:0/161625110 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8031c00 con 0x7f4eac072210 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4ea3fff640 10 monclient: dump: 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.348 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4eb32d6640 5 monclient: authenticate success, global_id 4299 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4eb32d6640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.353+0000 7f4eb32d6640 4 set_mon_vals no callback set 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7fd16ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd170071d00 0x7fd1700b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39076/0 (socket says 172.21.15.49:39076) 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7fd16ffff640 1 -- 172.21.15.49:0/2533115488 learned_addr learned my addr 172.21.15.49:0/2533115488 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7fd16ffff640 10 monclient(hunting): get_auth_request con 0x7fd170071d00 auth_method 0 2024-05-15T12:32:50.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7fd16ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7fd16ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7fd16ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f55eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f4eb32d6640 10 monclient: shutdown 2024-05-15T12:32:50.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f4eb32d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f4eb32d6640 1 -- 172.21.15.49:0/161625110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac072210 msgr2=0x7f4eac06fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f4eb32d6640 1 --2- 172.21.15.49:0/161625110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac072210 0x7f4eac06fae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4ea80097a0 tx=0x7f4ea802b8d0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f55eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f4eb32d6640 1 -- 172.21.15.49:0/161625110 shutdown_connections 2024-05-15T12:32:50.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f4eb32d6640 1 --2- 172.21.15.49:0/161625110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac072210 0x7f4eac06fae0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f4eb32d6640 1 -- 172.21.15.49:0/161625110 >> 172.21.15.49:0/161625110 conn(0x7f4eac0b4920 msgr2=0x7f4eac0b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f55eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f4eb32d6640 1 -- 172.21.15.49:0/161625110 shutdown_connections 2024-05-15T12:32:50.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7f4eb32d6640 1 -- 172.21.15.49:0/161625110 wait complete. 2024-05-15T12:32:50.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7fd16ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7fd16ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.354+0000 7fd16ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.355+0000 7f4eb32d6640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.355+0000 7f4eb32d6640 10 monclient: monmap: 2024-05-15T12:32:50.354 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.354 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:50.354 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:50.354 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:50.354 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.354 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.356+0000 7f4eb32d6640 1 librados: starting msgr at 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.356+0000 7f4eb32d6640 1 librados: starting objecter 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.356+0000 7f4eb32d6640 1 Processor -- start 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.356+0000 7f4eb32d6640 1 -- start start 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.356+0000 7f4eb32d6640 1 librados: setting wanted keys 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.356+0000 7f4eb32d6640 1 librados: calling monclient init 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.356+0000 7f4eb32d6640 10 monclient: init 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4eac072210 0x7f4eac133910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 10 monclient(hunting): picked mon.noname-b con 0x7f4eac072210 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:50.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac074560 0x7f4eac0aae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 10 monclient(hunting): picked mon.noname-a con 0x7f4eac074560 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:50.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f55ea7fc640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec140500 0x7f55ec0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4eac1420e0 0x7f4eac0ab350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 10 monclient(hunting): picked mon.noname-c con 0x7f4eac1420e0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac074c20 con 0x7f4eac074560 2024-05-15T12:32:50.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f8233fff640 1 -- 172.21.15.49:0/2453766991 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238039070 con 0x7f823c147210 2024-05-15T12:32:50.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f8233fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f8233fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f8233fff640 10 monclient: dump: 2024-05-15T12:32:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.359 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.359 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.359 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f8233fff640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f8233fff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f8233fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f82432cf640 5 monclient: authenticate success, global_id 4283 2024-05-15T12:32:50.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f82432cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:50.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac06f460 con 0x7f4eac072210 2024-05-15T12:32:50.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0aebf0 con 0x7f4eac1420e0 2024-05-15T12:32:50.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.357+0000 7f4eb32d6640 10 monclient(hunting): authenticate will time out at 1528.518910s 2024-05-15T12:32:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.359+0000 7f55eaffd640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-05-15T12:32:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.359+0000 7f55eaffd640 1 -- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec140500 msgr2=0x7f55ec0b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.359+0000 7f55eaffd640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec140500 0x7f55ec0b5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.359+0000 7f55eaffd640 1 -- 172.21.15.49:0/900168732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ec0b87e0 msgr2=0x7f55ec1d94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.359+0000 7f55eaffd640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ec0b87e0 0x7f55ec1d94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.359+0000 7f55eaffd640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.359+0000 7f55eaffd640 1 monclient: found mon.noname-a 2024-05-15T12:32:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.359+0000 7f55eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.359+0000 7f55eaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.359+0000 7f55eaffd640 1 -- 172.21.15.49:0/900168732 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec0b77c0 con 0x7f55d4004170 2024-05-15T12:32:50.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.363+0000 7fd16ffff640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2024-05-15T12:32:50.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.363+0000 7fd16ffff640 1 -- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 msgr2=0x7fd17012e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.363+0000 7fd16ffff640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 0x7fd17012e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.363+0000 7fd16ffff640 1 -- 172.21.15.49:0/2533115488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd17013ba30 msgr2=0x7fd1700b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.363+0000 7fd16ffff640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd17013ba30 0x7fd1700b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.363+0000 7fd16ffff640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.363+0000 7fd16ffff640 1 monclient: found mon.noname-a 2024-05-15T12:32:50.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.363+0000 7fd16ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.363+0000 7fd16ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.363+0000 7fd16ffff640 1 -- 172.21.15.49:0/2533115488 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd170135ef0 con 0x7fd170071d00 2024-05-15T12:32:50.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.368+0000 7fd16ffff640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd170071d00 0x7fd1700b1f60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd16000ba50 tx=0x7fd16000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.368+0000 7fd16cff9640 10 client.?.objecter ms_handle_connect 0x7fd170071d00 2024-05-15T12:32:50.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.368+0000 7fd16cff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:50.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.369+0000 7f4eb0871640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac074560 0x7f4eac0aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.369+0000 7f55eaffd640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 0x7f55ec142fd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f55e000b580 tx=0x7f55e000ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.369+0000 7f55d3fff640 10 client.?.objecter ms_handle_connect 0x7f55d4004170 2024-05-15T12:32:50.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.369+0000 7f55d3fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:50.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.373+0000 7f4eb0871640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac074560 0x7f4eac0aae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39082/0 (socket says 172.21.15.49:39082) 2024-05-15T12:32:50.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.373+0000 7f4eb0871640 1 -- 172.21.15.49:0/1729040121 learned_addr learned my addr 172.21.15.49:0/1729040121 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.373+0000 7f4eb0871640 10 monclient(hunting): get_auth_request con 0x7f4eac074560 auth_method 0 2024-05-15T12:32:50.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.373+0000 7f4eb0871640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.373+0000 7f4eb0871640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.373+0000 7f4eb0871640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 1 -- 172.21.15.49:0/2533115488 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160019070 con 0x7fd170071d00 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 10 monclient: dump: 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.366 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd176629640 5 monclient: authenticate success, global_id 4308 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd176629640 10 monclient: _renew_subs 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd176629640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd176629640 1 -- 172.21.15.49:0/2533115488 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1700b3130 con 0x7fd170071d00 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 1 -- 172.21.15.49:0/2533115488 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16000cc00 con 0x7fd170071d00 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 1 -- 172.21.15.49:0/2533115488 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160014440 con 0x7fd170071d00 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16cff9640 10 monclient: dump: 2024-05-15T12:32:50.367 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd16e7fc640 4 set_mon_vals no callback set 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd176629640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.376+0000 7fd176629640 10 monclient: _renew_subs 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.377+0000 7fd176629640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.377+0000 7fd176629640 1 -- 172.21.15.49:0/2533115488 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17012e5a0 con 0x7fd170071d00 2024-05-15T12:32:50.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.377+0000 7fd176629640 1 librados: init done 2024-05-15T12:32:50.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.377+0000 7f4eb0871640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.377+0000 7f4eb0871640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.377+0000 7f4eb0871640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.378+0000 7fd176629640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.378+0000 7fd176629640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.378+0000 7fd176629640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.378+0000 7fd176629640 1 -- 172.21.15.49:0/2533115488 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17004f0e0 con 0x7fd170071d00 2024-05-15T12:32:50.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.381+0000 7f4eb0871640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-05-15T12:32:50.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.381+0000 7f4eb0871640 1 -- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4eac1420e0 msgr2=0x7f4eac0ab350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.381+0000 7f4eb0871640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4eac1420e0 0x7f4eac0ab350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.381+0000 7f4eb0871640 1 -- 172.21.15.49:0/1729040121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4eac072210 msgr2=0x7f4eac133910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.381+0000 7f4eb0871640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4eac072210 0x7f4eac133910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.381+0000 7f4eb0871640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.381+0000 7f4eb0871640 1 monclient: found mon.noname-a 2024-05-15T12:32:50.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.381+0000 7f4eb0871640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.381+0000 7f4eb0871640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.381+0000 7f4eb0871640 1 -- 172.21.15.49:0/1729040121 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac134740 con 0x7f4eac074560 2024-05-15T12:32:50.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55d3fff640 1 -- 172.21.15.49:0/900168732 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e0014070 con 0x7f55d4004170 2024-05-15T12:32:50.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:50.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55d3fff640 10 monclient: dump: 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55d3fff640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55d3fff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55f18cf640 5 monclient: authenticate success, global_id 4305 2024-05-15T12:32:50.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55f18cf640 10 monclient: _renew_subs 2024-05-15T12:32:50.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55f18cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55f18cf640 1 -- 172.21.15.49:0/900168732 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ec0b62e0 con 0x7f55d4004170 2024-05-15T12:32:50.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55f18cf640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:50.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55f18cf640 10 monclient: _renew_subs 2024-05-15T12:32:50.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55f18cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55f18cf640 1 -- 172.21.15.49:0/900168732 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ec0b8c60 con 0x7f55d4004170 2024-05-15T12:32:50.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.387+0000 7f55f18cf640 1 librados: init done 2024-05-15T12:32:50.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.388+0000 7f55f18cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.388+0000 7f55f18cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.388+0000 7f55f18cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.388+0000 7f55f18cf640 1 -- 172.21.15.49:0/900168732 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ec0b8fd0 con 0x7f55d4004170 2024-05-15T12:32:50.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.520+0000 7fb0faffd640 1 --2- 172.21.15.49:0/1579199157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0f402fca0 0x7fb0f40c91c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4007050 tx=0x7fb0e802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=140606295ee55ab0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.522+0000 7f76e5ef3640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d8038db0 0x7f76d8009e00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f76e0068ca0 tx=0x7f76e006a350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.522+0000 7f76d67fc640 10 client.?.objecter ms_handle_connect 0x7f76d8038db0 2024-05-15T12:32:50.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.522+0000 7f76d67fc640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:50.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 1 -- 172.21.15.49:0/1579199157 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e802f810 con 0x7fb0f402fca0 2024-05-15T12:32:50.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:50.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 10 monclient: dump: 2024-05-15T12:32:50.517 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.517 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.517 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 1 -- 172.21.15.49:0/1579199157 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e802f9b0 con 0x7fb0f402fca0 2024-05-15T12:32:50.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 1 -- 172.21.15.49:0/1579199157 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e802fdc0 con 0x7fb0f402fca0 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb1009d8640 5 monclient: authenticate success, global_id 4336 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb1009d8640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb1009d8640 4 set_mon_vals no callback set 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.528+0000 7fb0fa7fc640 10 monclient: dump: 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.519 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.520 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.520 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.520 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.529+0000 7fb1009d8640 10 monclient: shutdown 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.529+0000 7fb1009d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.529+0000 7fb1009d8640 1 -- 172.21.15.49:0/1579199157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0f402fca0 msgr2=0x7fb0f40c91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.529+0000 7fb1009d8640 1 --2- 172.21.15.49:0/1579199157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0f402fca0 0x7fb0f40c91c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4007050 tx=0x7fb0e802b9f0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.529+0000 7fb1009d8640 1 -- 172.21.15.49:0/1579199157 shutdown_connections 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.529+0000 7fb1009d8640 1 --2- 172.21.15.49:0/1579199157 >> v2:172.21.15.49:3300/0 conn(0x7fb0f4032160 0x7fb0f40c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.529+0000 7fb1009d8640 1 --2- 172.21.15.49:0/1579199157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0f402fca0 0x7fb0f40c91c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.529+0000 7fb1009d8640 1 -- 172.21.15.49:0/1579199157 >> 172.21.15.49:0/1579199157 conn(0x7fb0f4005ee0 msgr2=0x7fb0f40062f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.529+0000 7fb1009d8640 1 -- 172.21.15.49:0/1579199157 shutdown_connections 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.529+0000 7fb1009d8640 1 -- 172.21.15.49:0/1579199157 wait complete. 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.530+0000 7fb1009d8640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.530+0000 7fb1009d8640 10 monclient: monmap: 2024-05-15T12:32:50.522 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.530+0000 7fb1009d8640 1 librados: starting msgr at 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.530+0000 7fb1009d8640 1 librados: starting objecter 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.530+0000 7fb1009d8640 1 Processor -- start 2024-05-15T12:32:50.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.530+0000 7fb1009d8640 1 -- start start 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.530+0000 7fb1009d8640 1 librados: setting wanted keys 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.530+0000 7fb1009d8640 1 librados: calling monclient init 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.530+0000 7fb1009d8640 10 monclient: init 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0f402fca0 0x7fb0f40cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 10 monclient(hunting): picked mon.noname-a con 0x7fb0f402fca0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0f4032160 0x7fb0f40a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 10 monclient(hunting): picked mon.noname-c con 0x7fb0f4032160 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0f4034c20 0x7fb0f40a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 10 monclient(hunting): picked mon.noname-b con 0x7fb0f4034c20 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f4031890 con 0x7fb0f402fca0 2024-05-15T12:32:50.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f4031710 con 0x7fb0f4034c20 2024-05-15T12:32:50.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40a2a40 con 0x7fb0f4032160 2024-05-15T12:32:50.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.531+0000 7fb1009d8640 10 monclient(hunting): authenticate will time out at 1528.692704s 2024-05-15T12:32:50.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 1 -- 172.21.15.49:0/4005751253 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e00731c0 con 0x7f76d8038db0 2024-05-15T12:32:50.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:50.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 10 monclient: dump: 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76e8157640 5 monclient: authenticate success, global_id 4342 2024-05-15T12:32:50.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 1 -- 172.21.15.49:0/4005751253 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e0073380 con 0x7f76d8038db0 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 1 -- 172.21.15.49:0/4005751253 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0093640 con 0x7f76d8038db0 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d67fc640 10 monclient: dump: 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.528 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76d7fff640 4 set_mon_vals no callback set 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76e8157640 10 monclient: _renew_subs 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76e8157640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.533+0000 7f76e8157640 1 -- 172.21.15.49:0/4005751253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d816a6a0 con 0x7f76d8038db0 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.534+0000 7f76e8157640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.534+0000 7f76e8157640 10 monclient: _renew_subs 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.534+0000 7f76e8157640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.534+0000 7f76e8157640 1 -- 172.21.15.49:0/4005751253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d816d0a0 con 0x7f76d8038db0 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.534+0000 7f76e8157640 1 librados: init done 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.535+0000 7f76e8157640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.535+0000 7f76e8157640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.535+0000 7f76e8157640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.535+0000 7f76e8157640 1 -- 172.21.15.49:0/4005751253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d8008630 con 0x7f76d8038db0 2024-05-15T12:32:50.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.547+0000 7f8d6bfff640 1 -- 172.21.15.49:0/801573055 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f8d7c005d10 con 0x7f8d840b5410 2024-05-15T12:32:50.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.547+0000 7f8d6bfff640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d7006ce20 0x7f8d7006f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.548+0000 7f8d827fc640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d7006ce20 0x7f8d7006f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.548+0000 7f8d827fc640 10 monclient: get_auth_request con 0x7f8d7006ce20 auth_method 0 2024-05-15T12:32:50.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.548+0000 7f8d6bfff640 10 client.4333.objecter ms_handle_connect 0x7f8d7006ce20 2024-05-15T12:32:50.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.548+0000 7f8d827fc640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d7006ce20 0x7f8d7006f2e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8d74005880 tx=0x7f8d74009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.556+0000 7f8d6bfff640 1 -- 172.21.15.49:0/801573055 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8d7c007df0 con 0x7f8d840b5410 2024-05-15T12:32:50.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.556+0000 7f8d6bfff640 10 client.4333.objecter ms_dispatch 0x7f8d84000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:50.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.556+0000 7f8d6bfff640 3 client.4333.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:50.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.556+0000 7f8d6bfff640 3 client.4333.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:50.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.556+0000 7f8d6bfff640 20 client.4333.objecter dump_active .. 0 homeless 2024-05-15T12:32:50.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.566+0000 7f8d6bfff640 1 -- 172.21.15.49:0/801573055 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8d7c022030 con 0x7f8d840b5410 2024-05-15T12:32:50.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.566+0000 7f8d6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.566+0000 7f8d6bfff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:50.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.580+0000 7f76d67fc640 1 -- 172.21.15.49:0/4005751253 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f76e00937e0 con 0x7f76d8038db0 2024-05-15T12:32:50.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.581+0000 7f76d67fc640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f76c406cdd0 0x7f76c406f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.581+0000 7f76e56f2640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f76c406cdd0 0x7f76c406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.581+0000 7f76e56f2640 10 monclient: get_auth_request con 0x7f76c406cdd0 auth_method 0 2024-05-15T12:32:50.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.582+0000 7f76e56f2640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f76c406cdd0 0x7f76c406f290 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f76d800cc60 tx=0x7f76dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.582+0000 7f76d67fc640 10 client.4342.objecter ms_handle_connect 0x7f76c406cdd0 2024-05-15T12:32:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.629+0000 7f92faffd640 1 -- 172.21.15.49:0/4059849635 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f92fc089180 con 0x7f930406fd00 2024-05-15T12:32:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.629+0000 7f92faffd640 10 client.4280.objecter ms_dispatch 0x7f9304000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.629+0000 7f92faffd640 3 client.4280.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.629+0000 7f92faffd640 3 client.4280.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.630+0000 7f92faffd640 20 client.4280.objecter dump_active .. 0 homeless 2024-05-15T12:32:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.630+0000 7f92faffd640 1 -- 172.21.15.49:0/4059849635 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f92fc059450 con 0x7f930406fd00 2024-05-15T12:32:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.630+0000 7f92faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.630+0000 7f92faffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.634+0000 7f8233fff640 1 -- 172.21.15.49:0/2453766991 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82380316f0 con 0x7f823c147210 2024-05-15T12:32:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.634+0000 7f8233fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.634+0000 7f8233fff640 1 -- 172.21.15.49:0/2453766991 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238031ac0 con 0x7f823c147210 2024-05-15T12:32:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.634+0000 7f82432cf640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.634+0000 7f82432cf640 4 set_mon_vals no callback set 2024-05-15T12:32:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.634+0000 7f8233fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.634+0000 7f8233fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.634+0000 7f8233fff640 10 monclient: dump: 2024-05-15T12:32:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.624 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.624 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.624 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.624 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.624 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.624 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.624 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.624 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.624 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.635+0000 7f82432cf640 10 monclient: shutdown 2024-05-15T12:32:50.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.635+0000 7f82432cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.635+0000 7f82432cf640 1 -- 172.21.15.49:0/2453766991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 msgr2=0x7f823c1438d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.635+0000 7f82432cf640 1 --2- 172.21.15.49:0/2453766991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c1438d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f82380097d0 tx=0x7f823802b790 comp rx=0 tx=0).stop 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.635+0000 7f82432cf640 1 -- 172.21.15.49:0/2453766991 shutdown_connections 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.635+0000 7f82432cf640 1 --2- 172.21.15.49:0/2453766991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c1438d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.635+0000 7f82432cf640 1 -- 172.21.15.49:0/2453766991 >> 172.21.15.49:0/2453766991 conn(0x7f823c066f00 msgr2=0x7f823c067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.635+0000 7f82432cf640 1 -- 172.21.15.49:0/2453766991 shutdown_connections 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.635+0000 7f82432cf640 1 -- 172.21.15.49:0/2453766991 wait complete. 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.636+0000 7f82432cf640 10 monclient: build_initial_monmap 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.636+0000 7f82432cf640 10 monclient: monmap: 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:50.626 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:50.627 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.627 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:50.627 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:50.627 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:50.627 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.636+0000 7fb0faffd640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0f4032160 0x7fb0f40a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.636+0000 7f82432cf640 1 librados: starting msgr at 2024-05-15T12:32:50.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.636+0000 7f82432cf640 1 librados: starting objecter 2024-05-15T12:32:50.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7f82432cf640 1 Processor -- start 2024-05-15T12:32:50.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7f82432cf640 1 -- start start 2024-05-15T12:32:50.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7f82432cf640 1 librados: setting wanted keys 2024-05-15T12:32:50.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7f82432cf640 1 librados: calling monclient init 2024-05-15T12:32:50.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7f82432cf640 10 monclient: init 2024-05-15T12:32:50.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7fb0faffd640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0f4032160 0x7fb0f40a0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46212/0 (socket says 172.21.15.49:46212) 2024-05-15T12:32:50.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7fb0faffd640 1 -- 172.21.15.49:0/3271586799 learned_addr learned my addr 172.21.15.49:0/3271586799 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7fb0faffd640 10 monclient(hunting): get_auth_request con 0x7fb0f4032160 auth_method 0 2024-05-15T12:32:50.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7fb0faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7fb0faffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7fb0faffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7fb0faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7fb0faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7fb0faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.637+0000 7f82432cf640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:50.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:32:50.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f823c0b53a0 0x7f823c1d3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 10 monclient(hunting): picked mon.noname-b con 0x7f823c0b53a0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:50.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f823c143e10 0x7f823c13e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 10 monclient(hunting): picked mon.noname-a con 0x7f823c143e10 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c13ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 10 monclient(hunting): picked mon.noname-c con 0x7f823c147210 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c06d250 con 0x7f823c143e10 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c0710d0 con 0x7f823c0b53a0 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823c1d29c0 con 0x7f823c147210 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:50.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.638+0000 7f82432cf640 10 monclient(hunting): authenticate will time out at 1528.799680s 2024-05-15T12:32:50.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.640+0000 7fb0faffd640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-05-15T12:32:50.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.640+0000 7fb0faffd640 1 -- 172.21.15.49:0/3271586799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0f4034c20 msgr2=0x7fb0f40a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.640+0000 7fb0faffd640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0f4034c20 0x7fb0f40a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.641+0000 7fb0faffd640 1 -- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0f402fca0 msgr2=0x7fb0f40cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.641+0000 7fb0faffd640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0f402fca0 0x7fb0f40cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.641+0000 7fb0faffd640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.641+0000 7fb0faffd640 1 monclient: found mon.noname-c 2024-05-15T12:32:50.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.641+0000 7fb0faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.641+0000 7fb0faffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.641+0000 7fb0faffd640 1 -- 172.21.15.49:0/3271586799 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f40a2d20 con 0x7fb0f4032160 2024-05-15T12:32:50.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.643+0000 7fb0faffd640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0f4032160 0x7fb0f40a0f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009960 tx=0x7fb0e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.643+0000 7fb0e7fff640 10 client.?.objecter ms_handle_connect 0x7fb0f4032160 2024-05-15T12:32:50.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.643+0000 7fb0e7fff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:50.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.643+0000 7f824186c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c13ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.645+0000 7f824186c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c13ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46224/0 (socket says 172.21.15.49:46224) 2024-05-15T12:32:50.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.645+0000 7f824186c640 1 -- 172.21.15.49:0/676202995 learned_addr learned my addr 172.21.15.49:0/676202995 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:50.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.645+0000 7f824186c640 10 monclient(hunting): get_auth_request con 0x7f823c147210 auth_method 0 2024-05-15T12:32:50.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.645+0000 7f824186c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:50.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.645+0000 7f824186c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:50.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.645+0000 7f824186c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:50.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.648+0000 7f824186c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:50.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.648+0000 7f824186c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:50.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.648+0000 7f824186c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:50.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.651+0000 7f824186c640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-05-15T12:32:50.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.651+0000 7f824186c640 1 -- 172.21.15.49:0/676202995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f823c0b53a0 msgr2=0x7f823c1d3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.651+0000 7f824186c640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f823c0b53a0 0x7f823c1d3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.651+0000 7f824186c640 1 -- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f823c143e10 msgr2=0x7f823c13e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.651+0000 7f824186c640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f823c143e10 0x7f823c13e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.651+0000 7f824186c640 10 monclient: _finish_hunting 0 2024-05-15T12:32:50.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.651+0000 7f824186c640 1 monclient: found mon.noname-c 2024-05-15T12:32:50.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.651+0000 7f824186c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:50.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.651+0000 7f824186c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.651+0000 7f824186c640 1 -- 172.21.15.49:0/676202995 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c13f200 con 0x7f823c147210 2024-05-15T12:32:50.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb0e7fff640 1 -- 172.21.15.49:0/3271586799 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8037810 con 0x7fb0f4032160 2024-05-15T12:32:50.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb0e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:50.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb0e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:50.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb0e7fff640 10 monclient: dump: 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb0e7fff640 10 monclient: _finish_auth 0 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb0e7fff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb0e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb1009d8640 5 monclient: authenticate success, global_id 4286 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb1009d8640 10 monclient: _renew_subs 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb1009d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb1009d8640 1 -- 172.21.15.49:0/3271586799 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f40a1ac0 con 0x7fb0f4032160 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb1009d8640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb1009d8640 10 monclient: _renew_subs 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb1009d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb1009d8640 1 -- 172.21.15.49:0/3271586799 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f40a1e30 con 0x7fb0f4032160 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.652+0000 7fb1009d8640 1 librados: init done 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.653+0000 7fb1009d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.653+0000 7fb1009d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.653+0000 7fb1009d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.653+0000 7fb1009d8640 1 -- 172.21.15.49:0/3271586799 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f4036bf0 con 0x7fb0f4032160 2024-05-15T12:32:50.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.672+0000 7fd16cff9640 1 -- 172.21.15.49:0/2533115488 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fd16001c070 con 0x7fd170071d00 2024-05-15T12:32:50.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.673+0000 7fd16cff9640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd15c06ce20 0x7fd15c06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:50.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.673+0000 7fd16f7fe640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd15c06ce20 0x7fd15c06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:50.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.673+0000 7fd16f7fe640 10 monclient: get_auth_request con 0x7fd15c06ce20 auth_method 0 2024-05-15T12:32:50.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.673+0000 7fd16cff9640 10 client.4308.objecter ms_handle_connect 0x7fd15c06ce20 2024-05-15T12:32:50.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.673+0000 7fd16f7fe640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd15c06ce20 0x7fd15c06f2e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd164000f60 tx=0x7fd1640079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.699+0000 7fd16cff9640 1 -- 172.21.15.49:0/2533115488 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd160056bf0 con 0x7fd170071d00 2024-05-15T12:32:50.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.700+0000 7fd16cff9640 10 client.4308.objecter ms_dispatch 0x7fd170000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:50.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.700+0000 7fd16cff9640 3 client.4308.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:50.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.700+0000 7fd16cff9640 3 client.4308.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:50.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.700+0000 7fd16cff9640 20 client.4308.objecter dump_active .. 0 homeless 2024-05-15T12:32:50.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.700+0000 7f4eb0871640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac074560 0x7f4eac0aae10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00b410 tx=0x7f4e9c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:50.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.700+0000 7fd16cff9640 1 -- 172.21.15.49:0/2533115488 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd160023020 con 0x7fd170071d00 2024-05-15T12:32:50.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.700+0000 7f4ea1ffb640 10 client.?.objecter ms_handle_connect 0x7f4eac074560 2024-05-15T12:32:50.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.700+0000 7f4ea1ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:50.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.700+0000 7fd16cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.700+0000 7fd16cff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:50.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.793+0000 7f8d69ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:32:50.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.793+0000 7f8d69ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:32:50.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.793+0000 7f8d69ffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.793+0000 7f8d69ffb640 1 -- 172.21.15.49:0/801573055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f8d84136e30 con 0x7f8d840b5410 2024-05-15T12:32:50.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.805+0000 7f76d67fc640 1 -- 172.21.15.49:0/4005751253 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f76e00d1270 con 0x7f76d8038db0 2024-05-15T12:32:50.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.805+0000 7f76d67fc640 10 client.4342.objecter ms_dispatch 0x7f76e0000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:50.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.805+0000 7f76d67fc640 3 client.4342.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:50.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.805+0000 7f76d67fc640 3 client.4342.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:50.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.805+0000 7f76d67fc640 20 client.4342.objecter dump_active .. 0 homeless 2024-05-15T12:32:50.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.814+0000 7f76d67fc640 1 -- 172.21.15.49:0/4005751253 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f76e00a2080 con 0x7f76d8038db0 2024-05-15T12:32:50.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.814+0000 7f76d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:50.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.814+0000 7f76d67fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:50.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.834+0000 7f8d6bfff640 1 -- 172.21.15.49:0/801573055 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8d7c010040 con 0x7f8d840b5410 2024-05-15T12:32:50.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.834+0000 7f8d6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:32:50.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.834+0000 7f8d6bfff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:50.824 INFO:teuthology.orchestra.run.smithi049.stdout:55834574853 2024-05-15T12:32:50.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.837+0000 7f8d69ffb640 10 librados: watch_flush enter 2024-05-15T12:32:50.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.838+0000 7f8d69ffb640 10 librados: watch_flush exit 2024-05-15T12:32:50.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.838+0000 7f8d69ffb640 20 client.4333.objecter shutdown clearing up homeless session... 2024-05-15T12:32:50.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.838+0000 7f8d69ffb640 10 client.4333.objecter successfully canceled tick 2024-05-15T12:32:50.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.838+0000 7f8d69ffb640 1 -- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d7006ce20 msgr2=0x7f8d7006f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.838+0000 7f8d69ffb640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d7006ce20 0x7f8d7006f2e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8d74005880 tx=0x7f8d74009040 comp rx=0 tx=0).stop 2024-05-15T12:32:50.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.838+0000 7f8d69ffb640 10 monclient: shutdown 2024-05-15T12:32:50.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.838+0000 7f8d69ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:50.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.838+0000 7f8d69ffb640 1 -- 172.21.15.49:0/801573055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d840b5410 msgr2=0x7f8d84136830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:50.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.838+0000 7f8d69ffb640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d840b5410 0x7f8d84136830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009f60 tx=0x7f8d7c009f90 comp rx=0 tx=0).stop 2024-05-15T12:32:50.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.838+0000 7f8d69ffb640 1 -- 172.21.15.49:0/801573055 shutdown_connections 2024-05-15T12:32:50.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.839+0000 7f8d69ffb640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d84145c00 0x7f8d841e46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.839+0000 7f8d69ffb640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d8413f990 0x7f8d84138f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.839+0000 7f8d69ffb640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d7006ce20 0x7f8d7006f2e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.839+0000 7f8d69ffb640 1 --2- 172.21.15.49:0/801573055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d840b5410 0x7f8d84136830 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:50.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.839+0000 7f8d69ffb640 1 -- 172.21.15.49:0/801573055 >> 172.21.15.49:0/801573055 conn(0x7f8d84066f00 msgr2=0x7f8d84135b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:50.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.839+0000 7f8d69ffb640 1 -- 172.21.15.49:0/801573055 shutdown_connections 2024-05-15T12:32:50.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.839+0000 7f8d69ffb640 1 -- 172.21.15.49:0/801573055 wait complete. 2024-05-15T12:32:50.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.839+0000 7f8d69ffb640 1 librados: shutdown 2024-05-15T12:32:50.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.851+0000 7f930c43a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:32:50.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.851+0000 7f930c43a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:32:50.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.851+0000 7f930c43a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:50.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.851+0000 7f930c43a640 1 -- 172.21.15.49:0/4059849635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f93041d80e0 con 0x7f930406fd00 2024-05-15T12:32:50.844 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2024-05-15T12:32:50.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.895+0000 7fd176629640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:32:50.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.895+0000 7fd176629640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:32:50.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.895+0000 7fd176629640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:50.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:50.895+0000 7fd176629640 1 -- 172.21.15.49:0/2533115488 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd1701d5960 con 0x7fd170071d00 2024-05-15T12:32:50.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.004+0000 7f76e8157640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:32:50.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.005+0000 7f76e8157640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:32:50.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.005+0000 7f76e8157640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:50.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.005+0000 7f76e8157640 1 -- 172.21.15.49:0/4005751253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f76d8005710 con 0x7f76d8038db0 2024-05-15T12:32:50.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.008+0000 7f76d67fc640 1 -- 172.21.15.49:0/4005751253 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f76d8005710 con 0x7f76d8038db0 2024-05-15T12:32:50.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.008+0000 7f76d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:32:50.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.008+0000 7f76d67fc640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:50.998 INFO:teuthology.orchestra.run.smithi049.stdout:60129542151 2024-05-15T12:32:51.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.010+0000 7f76e8157640 10 librados: watch_flush enter 2024-05-15T12:32:51.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.010+0000 7f76e8157640 10 librados: watch_flush exit 2024-05-15T12:32:51.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.010+0000 7f76e8157640 20 client.4342.objecter shutdown clearing up homeless session... 2024-05-15T12:32:51.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.010+0000 7f76e8157640 10 client.4342.objecter successfully canceled tick 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 -- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f76c406cdd0 msgr2=0x7f76c406f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f76c406cdd0 0x7f76c406f290 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f76d800cc60 tx=0x7f76dc009040 comp rx=0 tx=0).stop 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 10 monclient: shutdown 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 -- 172.21.15.49:0/4005751253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d8038db0 msgr2=0x7f76d8009e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d8038db0 0x7f76d8009e00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f76e0068ca0 tx=0x7f76e006a350 comp rx=0 tx=0).stop 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 -- 172.21.15.49:0/4005751253 shutdown_connections 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d800b050 0x7f76d816a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d800c860 0x7f76d800ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f76c406cdd0 0x7f76c406f290 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 --2- 172.21.15.49:0/4005751253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76d8038db0 0x7f76d8009e00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 -- 172.21.15.49:0/4005751253 >> 172.21.15.49:0/4005751253 conn(0x7f76d8006610 msgr2=0x7f76d80069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:51.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 -- 172.21.15.49:0/4005751253 shutdown_connections 2024-05-15T12:32:51.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 -- 172.21.15.49:0/4005751253 wait complete. 2024-05-15T12:32:51.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.011+0000 7f76e8157640 1 librados: shutdown 2024-05-15T12:32:51.014 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542151 for osd.6 2024-05-15T12:32:51.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.026+0000 7f824186c640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c13ecc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f822c00c440 tx=0x7f822c00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:51.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.026+0000 7f8231ffb640 10 client.?.objecter ms_handle_connect 0x7f823c147210 2024-05-15T12:32:51.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.026+0000 7f8231ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:51.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.034+0000 7f92faffd640 1 -- 172.21.15.49:0/4059849635 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f93041d80e0 con 0x7f930406fd00 2024-05-15T12:32:51.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.035+0000 7f92faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:32:51.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.035+0000 7f92faffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:51.024 INFO:teuthology.orchestra.run.smithi049.stdout:73014444039 2024-05-15T12:32:51.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 10 librados: watch_flush enter 2024-05-15T12:32:51.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 10 librados: watch_flush exit 2024-05-15T12:32:51.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 20 client.4280.objecter shutdown clearing up homeless session... 2024-05-15T12:32:51.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 10 client.4280.objecter successfully canceled tick 2024-05-15T12:32:51.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 -- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f92e806cdd0 msgr2=0x7f92e806f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f92e806cdd0 0x7f92e806f290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f92f4035000 tx=0x7f92f4008030 comp rx=0 tx=0).stop 2024-05-15T12:32:51.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 10 monclient: shutdown 2024-05-15T12:32:51.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:51.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 -- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406fd00 msgr2=0x7f93041d78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406fd00 0x7f93041d78f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00b080 tx=0x7f92fc00b540 comp rx=0 tx=0).stop 2024-05-15T12:32:51.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 -- 172.21.15.49:0/4059849635 shutdown_connections 2024-05-15T12:32:51.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f930406fd00 0x7f93041d78f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f930406ec10 0x7f93041415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f92e806cdd0 0x7f92e806f290 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 --2- 172.21.15.49:0/4059849635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f930406e830 0x7f9304134510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 -- 172.21.15.49:0/4059849635 >> 172.21.15.49:0/4059849635 conn(0x7f93040b3ca0 msgr2=0x7f93040b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:51.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 -- 172.21.15.49:0/4059849635 shutdown_connections 2024-05-15T12:32:51.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 -- 172.21.15.49:0/4059849635 wait complete. 2024-05-15T12:32:51.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.037+0000 7f930c43a640 1 librados: shutdown 2024-05-15T12:32:51.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.047+0000 7fb0e7fff640 1 -- 172.21.15.49:0/3271586799 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e8047070 con 0x7fb0f4032160 2024-05-15T12:32:51.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.047+0000 7fb0e7fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.047+0000 7fb0e7fff640 1 -- 172.21.15.49:0/3271586799 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8009470 con 0x7fb0f4032160 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.047+0000 7fb0e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.047+0000 7fb0e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.047+0000 7fb0e7fff640 10 monclient: dump: 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:51.038 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:51.038 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:51.038 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.047+0000 7fb0f9ffb640 4 set_mon_vals no callback set 2024-05-15T12:32:51.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.2 2024-05-15T12:32:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.048+0000 7fb0e7fff640 1 -- 172.21.15.49:0/3271586799 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fb0e8040050 con 0x7fb0f4032160 2024-05-15T12:32:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.049+0000 7fb0e7fff640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb0e006cdd0 0x7fb0e006f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:51.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.049+0000 7fb0fb7fe640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb0e006cdd0 0x7fb0e006f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:51.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.049+0000 7fb0fb7fe640 10 monclient: get_auth_request con 0x7fb0e006cdd0 auth_method 0 2024-05-15T12:32:51.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.049+0000 7fb0e7fff640 10 client.4286.objecter ms_handle_connect 0x7fb0e006cdd0 2024-05-15T12:32:51.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.049+0000 7fb0fb7fe640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb0e006cdd0 0x7fb0e006f290 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb0f00010e0 tx=0x7fb0f0007950 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:51.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.067+0000 7fb0e7fff640 1 -- 172.21.15.49:0/3271586799 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb0e80ab8b0 con 0x7fb0f4032160 2024-05-15T12:32:51.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.067+0000 7fb0e7fff640 10 client.4286.objecter ms_dispatch 0x7fb0fc000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:51.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.067+0000 7fb0e7fff640 3 client.4286.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:51.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.067+0000 7fb0e7fff640 3 client.4286.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:51.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.067+0000 7fb0e7fff640 20 client.4286.objecter dump_active .. 0 homeless 2024-05-15T12:32:51.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.067+0000 7fb0e7fff640 1 -- 172.21.15.49:0/3271586799 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb0f4036bf0 con 0x7fb0f4032160 2024-05-15T12:32:51.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.067+0000 7fb0e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:51.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.067+0000 7fb0e7fff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:51.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 1 -- 172.21.15.49:0/676202995 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c017070 con 0x7f823c147210 2024-05-15T12:32:51.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:51.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:51.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 10 monclient: dump: 2024-05-15T12:32:51.065 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:51.065 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.065 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.065 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 10 monclient: _finish_auth 0 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 1 -- 172.21.15.49:0/676202995 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822c0035f0 con 0x7f823c147210 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f82432cf640 5 monclient: authenticate success, global_id 4289 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f82432cf640 10 monclient: _renew_subs 2024-05-15T12:32:51.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f82432cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f82432cf640 1 -- 172.21.15.49:0/676202995 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823c1d38c0 con 0x7f823c147210 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 1 -- 172.21.15.49:0/676202995 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c0039e0 con 0x7f823c147210 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f8231ffb640 10 monclient: dump: 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:51.067 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:51.068 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:51.068 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f82337fe640 4 set_mon_vals no callback set 2024-05-15T12:32:51.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f82432cf640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:51.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f82432cf640 10 monclient: _renew_subs 2024-05-15T12:32:51.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f82432cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:51.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f82432cf640 1 -- 172.21.15.49:0/676202995 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f823c13f640 con 0x7f823c147210 2024-05-15T12:32:51.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.075+0000 7f82432cf640 1 librados: init done 2024-05-15T12:32:51.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f55d3fff640 1 -- 172.21.15.49:0/900168732 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e0010890 con 0x7f55d4004170 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f55d3fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f55d3fff640 1 -- 172.21.15.49:0/900168732 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e001e060 con 0x7f55d4004170 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f55d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f55d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f55d3fff640 10 monclient: dump: 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:51.069 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f55e97fa640 4 set_mon_vals no callback set 2024-05-15T12:32:51.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f82432cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:51.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f82432cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:51.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f82432cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:51.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.077+0000 7f55d3fff640 1 -- 172.21.15.49:0/900168732 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f55e0017070 con 0x7f55d4004170 2024-05-15T12:32:51.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.076+0000 7f82432cf640 1 -- 172.21.15.49:0/676202995 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f823c04f0e0 con 0x7f823c147210 2024-05-15T12:32:51.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.077+0000 7f55d3fff640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f55d806cdd0 0x7f55d806f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:51.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.077+0000 7f55ea7fc640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f55d806cdd0 0x7f55d806f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:51.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.077+0000 7f55ea7fc640 10 monclient: get_auth_request con 0x7f55d806cdd0 auth_method 0 2024-05-15T12:32:51.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.078+0000 7f55ea7fc640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f55d806cdd0 0x7f55d806f290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f55dc034000 tx=0x7f55dc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:51.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.078+0000 7f55d3fff640 10 client.4305.objecter ms_handle_connect 0x7f55d806cdd0 2024-05-15T12:32:51.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.093+0000 7f55d3fff640 1 -- 172.21.15.49:0/900168732 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f55e0088990 con 0x7f55d4004170 2024-05-15T12:32:51.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.093+0000 7f55d3fff640 10 client.4305.objecter ms_dispatch 0x7f55ec000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:51.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.093+0000 7f55d3fff640 3 client.4305.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:51.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.093+0000 7f55d3fff640 3 client.4305.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:51.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.093+0000 7f55d3fff640 20 client.4305.objecter dump_active .. 0 homeless 2024-05-15T12:32:51.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.093+0000 7f55d3fff640 1 -- 172.21.15.49:0/900168732 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f55e00592b0 con 0x7f55d4004170 2024-05-15T12:32:51.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.093+0000 7f55d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:51.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.093+0000 7f55d3fff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:51.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.102+0000 7f8231ffb640 1 -- 172.21.15.49:0/676202995 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f822c01a070 con 0x7f823c147210 2024-05-15T12:32:51.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.103+0000 7f8231ffb640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f822006ce20 0x7f822006f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:51.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.103+0000 7f824106b640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f822006ce20 0x7f822006f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:51.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.103+0000 7f824106b640 10 monclient: get_auth_request con 0x7f822006ce20 auth_method 0 2024-05-15T12:32:51.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.103+0000 7f8231ffb640 10 client.4289.objecter ms_handle_connect 0x7f822006ce20 2024-05-15T12:32:51.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.103+0000 7f824106b640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f822006ce20 0x7f822006f2e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82340012e0 tx=0x7f8234001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:51.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.114+0000 7fd16cff9640 1 -- 172.21.15.49:0/2533115488 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd160059250 con 0x7fd170071d00 2024-05-15T12:32:51.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.114+0000 7fd16cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:32:51.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.114+0000 7fd16cff9640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:51.105 INFO:teuthology.orchestra.run.smithi049.stdout:81604378631 2024-05-15T12:32:51.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.116+0000 7fd176629640 10 librados: watch_flush enter 2024-05-15T12:32:51.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.116+0000 7fd176629640 10 librados: watch_flush exit 2024-05-15T12:32:51.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.116+0000 7fd176629640 20 client.4308.objecter shutdown clearing up homeless session... 2024-05-15T12:32:51.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.116+0000 7fd176629640 10 client.4308.objecter successfully canceled tick 2024-05-15T12:32:51.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.116+0000 7fd176629640 1 -- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd15c06ce20 msgr2=0x7fd15c06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd15c06ce20 0x7fd15c06f2e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd164000f60 tx=0x7fd1640079b0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 10 monclient: shutdown 2024-05-15T12:32:51.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:51.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 -- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd170071d00 msgr2=0x7fd1700b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd170071d00 0x7fd1700b1f60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd16000ba50 tx=0x7fd16000bf10 comp rx=0 tx=0).stop 2024-05-15T12:32:51.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 -- 172.21.15.49:0/2533115488 shutdown_connections 2024-05-15T12:32:51.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17013ed00 0x7fd17012e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd15c06ce20 0x7fd15c06f2e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd17013ba30 0x7fd1700b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 --2- 172.21.15.49:0/2533115488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd170071d00 0x7fd1700b1f60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 -- 172.21.15.49:0/2533115488 >> 172.21.15.49:0/2533115488 conn(0x7fd1700b3e10 msgr2=0x7fd1701348d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:51.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 -- 172.21.15.49:0/2533115488 shutdown_connections 2024-05-15T12:32:51.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 -- 172.21.15.49:0/2533115488 wait complete. 2024-05-15T12:32:51.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.117+0000 7fd176629640 1 librados: shutdown 2024-05-15T12:32:51.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 1 -- 172.21.15.49:0/1729040121 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c015020 con 0x7f4eac074560 2024-05-15T12:32:51.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 10 monclient: dump: 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 10 monclient: _finish_auth 0 2024-05-15T12:32:51.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 1 -- 172.21.15.49:0/1729040121 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9c011830 con 0x7f4eac074560 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4eb32d6640 5 monclient: authenticate success, global_id 4311 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4eb32d6640 10 monclient: _renew_subs 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 1 -- 172.21.15.49:0/1729040121 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c011c20 con 0x7f4eac074560 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4eb32d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4eb32d6640 1 -- 172.21.15.49:0/1729040121 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac1349b0 con 0x7f4eac074560 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea37fe640 4 set_mon_vals no callback set 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.124+0000 7f4ea1ffb640 10 monclient: dump: 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:51.116 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.125+0000 7f4eb32d6640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.125+0000 7f4eb32d6640 10 monclient: _renew_subs 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.125+0000 7f4eb32d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.125+0000 7f4eb32d6640 1 -- 172.21.15.49:0/1729040121 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac134f70 con 0x7f4eac074560 2024-05-15T12:32:51.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.125+0000 7f4eb32d6640 1 librados: init done 2024-05-15T12:32:51.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.126+0000 7f4eb32d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:51.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.126+0000 7f4eb32d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:51.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.126+0000 7f4eb32d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:51.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.126+0000 7f4eb32d6640 1 -- 172.21.15.49:0/1729040121 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac04f0e0 con 0x7f4eac074560 2024-05-15T12:32:51.120 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378633 got 81604378631 for osd.5 2024-05-15T12:32:51.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.267+0000 7fb1009d8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:32:51.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.267+0000 7fb1009d8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:32:51.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.267+0000 7fb1009d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:51.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.267+0000 7fb1009d8640 1 -- 172.21.15.49:0/3271586799 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb0f40052c0 con 0x7fb0f4032160 2024-05-15T12:32:51.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.313+0000 7f55f18cf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:32:51.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.313+0000 7f55f18cf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:32:51.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.313+0000 7f55f18cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:51.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.313+0000 7f55f18cf640 1 -- 172.21.15.49:0/900168732 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f55ec146ad0 con 0x7f55d4004170 2024-05-15T12:32:51.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.423+0000 7f8231ffb640 1 -- 172.21.15.49:0/676202995 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f822c0561a0 con 0x7f823c147210 2024-05-15T12:32:51.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.423+0000 7f8231ffb640 10 client.4289.objecter ms_dispatch 0x7f823c000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:51.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.424+0000 7f8231ffb640 3 client.4289.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:51.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.424+0000 7f8231ffb640 3 client.4289.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:51.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.424+0000 7f8231ffb640 20 client.4289.objecter dump_active .. 0 homeless 2024-05-15T12:32:51.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.424+0000 7f8231ffb640 1 -- 172.21.15.49:0/676202995 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f822c021020 con 0x7f823c147210 2024-05-15T12:32:51.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.424+0000 7f8231ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:51.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.424+0000 7f8231ffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:51.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.493+0000 7fb0e7fff640 1 -- 172.21.15.49:0/3271586799 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb0e8033030 con 0x7fb0f4032160 2024-05-15T12:32:51.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.493+0000 7fb0e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:32:51.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.493+0000 7fb0e7fff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:51.484 INFO:teuthology.orchestra.run.smithi049.stdout:90194313224 2024-05-15T12:32:51.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 10 librados: watch_flush enter 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 10 librados: watch_flush exit 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 20 client.4286.objecter shutdown clearing up homeless session... 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 10 client.4286.objecter successfully canceled tick 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 -- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb0e006cdd0 msgr2=0x7fb0e006f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb0e006cdd0 0x7fb0e006f290 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb0f00010e0 tx=0x7fb0f0007950 comp rx=0 tx=0).stop 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 10 monclient: shutdown 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 -- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0f4032160 msgr2=0x7fb0f40a0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0f4032160 0x7fb0f40a0f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009960 tx=0x7fb0e80046c0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 -- 172.21.15.49:0/3271586799 shutdown_connections 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb0e006cdd0 0x7fb0e006f290 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0f4034c20 0x7fb0f40a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0f4032160 0x7fb0f40a0f40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 --2- 172.21.15.49:0/3271586799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0f402fca0 0x7fb0f40cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 -- 172.21.15.49:0/3271586799 >> 172.21.15.49:0/3271586799 conn(0x7fb0f4005ee0 msgr2=0x7fb0f40d6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:51.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 -- 172.21.15.49:0/3271586799 shutdown_connections 2024-05-15T12:32:51.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 -- 172.21.15.49:0/3271586799 wait complete. 2024-05-15T12:32:51.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.495+0000 7fb1009d8640 1 librados: shutdown 2024-05-15T12:32:51.496 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313224 for osd.1 2024-05-15T12:32:51.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.538+0000 7f4ea1ffb640 1 -- 172.21.15.49:0/1729040121 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f4e9c018020 con 0x7f4eac074560 2024-05-15T12:32:51.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.538+0000 7f4ea1ffb640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4e9006ce00 0x7f4e9006f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:51.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.538+0000 7f4eb1072640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4e9006ce00 0x7f4e9006f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:51.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.538+0000 7f4eb1072640 10 monclient: get_auth_request con 0x7f4e9006ce00 auth_method 0 2024-05-15T12:32:51.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.539+0000 7f4ea1ffb640 10 client.4311.objecter ms_handle_connect 0x7f4e9006ce00 2024-05-15T12:32:51.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.539+0000 7f4eb1072640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4e9006ce00 0x7f4e9006f2c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8036000 tx=0x7f4ea8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:51.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.558+0000 7f4ea1ffb640 1 -- 172.21.15.49:0/1729040121 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4e9c022e50 con 0x7f4eac074560 2024-05-15T12:32:51.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.558+0000 7f4ea1ffb640 10 client.4311.objecter ms_dispatch 0x7f4eac000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:51.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.558+0000 7f4ea1ffb640 3 client.4311.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:51.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.558+0000 7f4ea1ffb640 3 client.4311.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:51.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.558+0000 7f4ea1ffb640 20 client.4311.objecter dump_active .. 0 homeless 2024-05-15T12:32:51.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.558+0000 7f4ea1ffb640 1 -- 172.21.15.49:0/1729040121 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4e9c01f070 con 0x7f4eac074560 2024-05-15T12:32:51.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.558+0000 7f4ea1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:51.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.558+0000 7f4ea1ffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:51.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.559+0000 7f55d3fff640 1 -- 172.21.15.49:0/900168732 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f55e0058bb0 con 0x7f55d4004170 2024-05-15T12:32:51.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.559+0000 7f55d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:32:51.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.559+0000 7f55d3fff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:51.550 INFO:teuthology.orchestra.run.smithi049.stdout:81604378632 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 10 librados: watch_flush enter 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 10 librados: watch_flush exit 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 20 client.4305.objecter shutdown clearing up homeless session... 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 10 client.4305.objecter successfully canceled tick 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 -- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f55d806cdd0 msgr2=0x7f55d806f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f55d806cdd0 0x7f55d806f290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f55dc034000 tx=0x7f55dc008030 comp rx=0 tx=0).stop 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 10 monclient: shutdown 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 -- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 msgr2=0x7f55ec142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 0x7f55ec142fd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f55e000b580 tx=0x7f55e000ba40 comp rx=0 tx=0).stop 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 -- 172.21.15.49:0/900168732 shutdown_connections 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f55d806cdd0 0x7f55d806f290 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ec0b87e0 0x7f55ec1d94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55ec140500 0x7f55ec0b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 --2- 172.21.15.49:0/900168732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55d4004170 0x7f55ec142fd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 -- 172.21.15.49:0/900168732 >> 172.21.15.49:0/900168732 conn(0x7f55ec066f00 msgr2=0x7f55ec1382b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:51.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 -- 172.21.15.49:0/900168732 shutdown_connections 2024-05-15T12:32:51.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 -- 172.21.15.49:0/900168732 wait complete. 2024-05-15T12:32:51.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.562+0000 7f55f18cf640 1 librados: shutdown 2024-05-15T12:32:51.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378634 got 81604378632 for osd.7 2024-05-15T12:32:51.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.618+0000 7f82432cf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:32:51.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.618+0000 7f82432cf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:32:51.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.618+0000 7f82432cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:51.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.618+0000 7f82432cf640 1 -- 172.21.15.49:0/676202995 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f823c1d0590 con 0x7f823c147210 2024-05-15T12:32:51.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.624+0000 7f8231ffb640 1 -- 172.21.15.49:0/676202995 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f822c058800 con 0x7f823c147210 2024-05-15T12:32:51.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.624+0000 7f8231ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:32:51.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.624+0000 7f8231ffb640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:51.614 INFO:teuthology.orchestra.run.smithi049.stdout:73014444039 2024-05-15T12:32:51.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 10 librados: watch_flush enter 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 10 librados: watch_flush exit 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 20 client.4289.objecter shutdown clearing up homeless session... 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 10 client.4289.objecter successfully canceled tick 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 -- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f822006ce20 msgr2=0x7f822006f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f822006ce20 0x7f822006f2e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82340012e0 tx=0x7f8234001630 comp rx=0 tx=0).stop 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 10 monclient: shutdown 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 -- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 msgr2=0x7f823c13ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c13ecc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f822c00c440 tx=0x7f822c00c900 comp rx=0 tx=0).stop 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 -- 172.21.15.49:0/676202995 shutdown_connections 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f823c147210 0x7f823c13ecc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f823c143e10 0x7f823c13e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f822006ce20 0x7f822006f2e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 --2- 172.21.15.49:0/676202995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f823c0b53a0 0x7f823c1d3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 -- 172.21.15.49:0/676202995 >> 172.21.15.49:0/676202995 conn(0x7f823c066f00 msgr2=0x7f823c142fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:51.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 -- 172.21.15.49:0/676202995 shutdown_connections 2024-05-15T12:32:51.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 -- 172.21.15.49:0/676202995 wait complete. 2024-05-15T12:32:51.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.626+0000 7f82432cf640 1 librados: shutdown 2024-05-15T12:32:51.628 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.0 2024-05-15T12:32:51.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.745+0000 7f4eb32d6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-15T12:32:51.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.745+0000 7f4eb32d6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-15T12:32:51.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.745+0000 7f4eb32d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:51.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.745+0000 7f4eb32d6640 1 -- 172.21.15.49:0/1729040121 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4eac0ad6f0 con 0x7f4eac074560 2024-05-15T12:32:51.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.749+0000 7f4ea1ffb640 1 -- 172.21.15.49:0/1729040121 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e9c059180 con 0x7f4eac074560 2024-05-15T12:32:51.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.749+0000 7f4ea1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-15T12:32:51.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.749+0000 7f4ea1ffb640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:51.739 INFO:teuthology.orchestra.run.smithi049.stdout:64424509446 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 10 librados: watch_flush enter 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 10 librados: watch_flush exit 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 20 client.4311.objecter shutdown clearing up homeless session... 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 10 client.4311.objecter successfully canceled tick 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 -- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4e9006ce00 msgr2=0x7f4e9006f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4e9006ce00 0x7f4e9006f2c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8036000 tx=0x7f4ea8030000 comp rx=0 tx=0).stop 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 10 monclient: shutdown 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 -- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac074560 msgr2=0x7f4eac0aae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac074560 0x7f4eac0aae10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00b410 tx=0x7f4e9c00b8d0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 -- 172.21.15.49:0/1729040121 shutdown_connections 2024-05-15T12:32:51.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4eac1420e0 0x7f4eac0ab350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4eac074560 0x7f4eac0aae10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4e9006ce00 0x7f4e9006f2c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 --2- 172.21.15.49:0/1729040121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4eac072210 0x7f4eac133910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 -- 172.21.15.49:0/1729040121 >> 172.21.15.49:0/1729040121 conn(0x7f4eac0b4920 msgr2=0x7f4eac0b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:51.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 -- 172.21.15.49:0/1729040121 shutdown_connections 2024-05-15T12:32:51.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 -- 172.21.15.49:0/1729040121 wait complete. 2024-05-15T12:32:51.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.751+0000 7f4eb32d6640 1 librados: shutdown 2024-05-15T12:32:51.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509446 for osd.3 2024-05-15T12:32:51.752 DEBUG:teuthology.parallel:result is None 2024-05-15T12:32:51.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-15T12:32:51.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.961+0000 7fa2c34be640 10 monclient: get_monmap_and_config 2024-05-15T12:32:51.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.961+0000 7fa2c34be640 10 monclient: build_initial_monmap 2024-05-15T12:32:51.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.961+0000 7fa2c34be640 10 monclient: monmap: 2024-05-15T12:32:51.951 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:51.951 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.951 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:51.962946+0000 2024-05-15T12:32:51.952 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:51.962946+0000 2024-05-15T12:32:51.952 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:51.952 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.952 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:51.952 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:51.952 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:51.952 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:51.952 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:51.952 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.962+0000 7fa2c34be640 1 Processor -- start 2024-05-15T12:32:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.962+0000 7fa2c34be640 1 -- start start 2024-05-15T12:32:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.962+0000 7fa2c34be640 10 monclient: init 2024-05-15T12:32:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.962+0000 7fa2c34be640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.962+0000 7fa2c34be640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-05-15T12:32:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.962+0000 7fa2c34be640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc0b5410 0x7fa2bc0a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.962+0000 7fa2c34be640 10 monclient(hunting): picked mon.noname-b con 0x7fa2bc0b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c34be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa2bc13f990 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c34be640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa2bc145c00 0x7fa2bc146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c34be640 10 monclient(hunting): picked mon.noname-a con 0x7fa2bc145c00 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c34be640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc0ab470 con 0x7fa2bc13f990 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c34be640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c34be640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2bc0b57f0 con 0x7fa2bc145c00 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c34be640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c34be640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0a9ea0 con 0x7fa2bc0b5410 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c34be640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c34be640 10 monclient(hunting): authenticate will time out at 1530.125518s 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c125a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc0b5410 0x7fa2bc0a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.963+0000 7fa2c1a5b640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa2bc145c00 0x7fa2bc146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c125a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc0b5410 0x7fa2bc0a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46240/0 (socket says 172.21.15.49:46240) 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c125a640 1 -- 172.21.15.49:0/630814118 learned_addr learned my addr 172.21.15.49:0/630814118 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c125a640 10 monclient(hunting): get_auth_request con 0x7fa2bc0b5410 auth_method 0 2024-05-15T12:32:51.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c125a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c125a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c125a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c1a5b640 10 monclient(hunting): get_auth_request con 0x7fa2bc145c00 auth_method 0 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c1a5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c1a5b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c1a5b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c125a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c125a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c125a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c1a5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c1a5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:51.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.964+0000 7fa2c1a5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:51.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.966+0000 7fa2c125a640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-05-15T12:32:51.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.966+0000 7fa2c125a640 1 -- 172.21.15.49:0/630814118 >> v2:172.21.15.49:3300/0 conn(0x7fa2bc145c00 msgr2=0x7fa2bc146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:51.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.966+0000 7fa2c125a640 1 --2- 172.21.15.49:0/630814118 >> v2:172.21.15.49:3300/0 conn(0x7fa2bc145c00 0x7fa2bc146070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.966+0000 7fa2c125a640 1 -- 172.21.15.49:0/630814118 >> v1:172.21.15.49:6789/0 conn(0x7fa2bc13f990 legacy=0x7fa2bc13fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.966+0000 7fa2c125a640 10 monclient: _finish_hunting 0 2024-05-15T12:32:51.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.966+0000 7fa2c125a640 1 monclient: found mon.noname-b 2024-05-15T12:32:51.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.966+0000 7fa2c125a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:51.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.966+0000 7fa2c125a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:51.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.966+0000 7fa2c125a640 1 -- 172.21.15.49:0/630814118 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc0aa0c0 con 0x7fa2bc0b5410 2024-05-15T12:32:51.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.969+0000 7fa2c125a640 1 --2- 172.21.15.49:0/630814118 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc0b5410 0x7fa2bc0a9960 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009870 tx=0x7fa2b802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0d9eb15a24cb74f server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:51.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 1 -- 172.21.15.49:0/630814118 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8035020 con 0x7fa2bc0b5410 2024-05-15T12:32:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 10 monclient: dump: 2024-05-15T12:32:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.963 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:51.964 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:51.964 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:51.964 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 10 monclient: _finish_auth 0 2024-05-15T12:32:51.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:51.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:51.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 1 -- 172.21.15.49:0/630814118 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b8031640 con 0x7fa2bc0b5410 2024-05-15T12:32:51.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:51.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 1 -- 172.21.15.49:0/630814118 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b802d040 con 0x7fa2bc0b5410 2024-05-15T12:32:51.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2c34be640 5 monclient: authenticate success, global_id 4292 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2c34be640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2c34be640 4 set_mon_vals no callback set 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.972+0000 7fa2b3fff640 10 monclient: dump: 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.965 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:51.966 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:51.966 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:51.966 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.973+0000 7fa2c34be640 10 monclient: shutdown 2024-05-15T12:32:51.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.973+0000 7fa2c34be640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:51.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.973+0000 7fa2c34be640 1 -- 172.21.15.49:0/630814118 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc0b5410 msgr2=0x7fa2bc0a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.973+0000 7fa2c34be640 1 --2- 172.21.15.49:0/630814118 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc0b5410 0x7fa2bc0a9960 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009870 tx=0x7fa2b802b8f0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.973+0000 7fa2c34be640 1 -- 172.21.15.49:0/630814118 shutdown_connections 2024-05-15T12:32:51.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.973+0000 7fa2c34be640 1 --2- 172.21.15.49:0/630814118 >> v2:172.21.15.49:3300/0 conn(0x7fa2bc145c00 0x7fa2bc146070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.973+0000 7fa2c34be640 1 --2- 172.21.15.49:0/630814118 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc0b5410 0x7fa2bc0a9960 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.973+0000 7fa2c34be640 1 -- 172.21.15.49:0/630814118 >> 172.21.15.49:0/630814118 conn(0x7fa2bc066f00 msgr2=0x7fa2bc067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.973+0000 7fa2c34be640 1 -- 172.21.15.49:0/630814118 shutdown_connections 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.973+0000 7fa2c34be640 1 -- 172.21.15.49:0/630814118 wait complete. 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.974+0000 7fa2c34be640 10 monclient: build_initial_monmap 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.974+0000 7fa2c34be640 10 monclient: monmap: 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:51.971 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.974+0000 7fa2c34be640 1 librados: starting msgr at 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.974+0000 7fa2c34be640 1 librados: starting objecter 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.974+0000 7fa2c34be640 1 Processor -- start 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 1 -- start start 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 1 librados: setting wanted keys 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 1 librados: calling monclient init 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient: init 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2bc0b5410 0x7fa2bc1484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:51.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient(hunting): picked mon.noname-a con 0x7fa2bc0b5410 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2bc13f990 0x7fa2bc147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient(hunting): picked mon.noname-b con 0x7fa2bc13f990 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc145c00 0x7fa2bc142d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient(hunting): picked mon.noname-c con 0x7fa2bc145c00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0b4aa0 con 0x7fa2bc0b5410 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0b4920 con 0x7fa2bc13f990 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc14cf30 con 0x7fa2bc145c00 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.975+0000 7fa2c34be640 10 monclient(hunting): authenticate will time out at 1530.138253s 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.976+0000 7fa2c125a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2bc0b5410 0x7fa2bc1484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.976+0000 7fa2c0a59640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2bc13f990 0x7fa2bc147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:51.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.976+0000 7fa2c1a5b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc145c00 0x7fa2bc142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c1a5b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc145c00 0x7fa2bc142d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46250/0 (socket says 172.21.15.49:46250) 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c1a5b640 1 -- 172.21.15.49:0/736444690 learned_addr learned my addr 172.21.15.49:0/736444690 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c1a5b640 10 monclient(hunting): get_auth_request con 0x7fa2bc145c00 auth_method 0 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c1a5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c1a5b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c1a5b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c125a640 10 monclient(hunting): get_auth_request con 0x7fa2bc0b5410 auth_method 0 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c125a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c125a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c125a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c0a59640 10 monclient(hunting): get_auth_request con 0x7fa2bc13f990 auth_method 0 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c0a59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c0a59640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:51.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c0a59640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:51.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c1a5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:51.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c1a5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:51.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c1a5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:51.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c125a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:51.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c125a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:51.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c125a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:51.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c0a59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:51.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c0a59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:51.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.977+0000 7fa2c0a59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:51.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.980+0000 7fa2c1a5b640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-05-15T12:32:51.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.980+0000 7fa2c1a5b640 1 -- 172.21.15.49:0/736444690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2bc13f990 msgr2=0x7fa2bc147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.980+0000 7fa2c1a5b640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2bc13f990 0x7fa2bc147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.980+0000 7fa2c1a5b640 1 -- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2bc0b5410 msgr2=0x7fa2bc1484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:51.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.980+0000 7fa2c1a5b640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2bc0b5410 0x7fa2bc1484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:51.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.980+0000 7fa2c1a5b640 10 monclient: _finish_hunting 0 2024-05-15T12:32:51.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.980+0000 7fa2c1a5b640 1 monclient: found mon.noname-c 2024-05-15T12:32:51.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.980+0000 7fa2c1a5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.980+0000 7fa2c1a5b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.980+0000 7fa2c1a5b640 1 -- 172.21.15.49:0/736444690 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc149660 con 0x7fa2bc145c00 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.983+0000 7fa2c1a5b640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc145c00 0x7fa2bc142d10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400ec10 tx=0x7fa2b400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.983+0000 7fa2b1ffb640 10 client.?.objecter ms_handle_connect 0x7fa2bc145c00 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.983+0000 7fa2b1ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 1 -- 172.21.15.49:0/736444690 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4018020 con 0x7fa2bc145c00 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 10 monclient: dump: 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.978 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 10 monclient: _finish_auth 0 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 1 -- 172.21.15.49:0/736444690 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b400cd20 con 0x7fa2bc145c00 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2c34be640 5 monclient: authenticate success, global_id 4295 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2c34be640 10 monclient: _renew_subs 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2c34be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2c34be640 1 -- 172.21.15.49:0/736444690 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc149940 con 0x7fa2bc145c00 2024-05-15T12:32:51.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 1 -- 172.21.15.49:0/736444690 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b40146c0 con 0x7fa2bc145c00 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b1ffb640 10 monclient: dump: 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:51.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.987+0000 7fa2c34be640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:51.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.987+0000 7fa2c34be640 10 monclient: _renew_subs 2024-05-15T12:32:51.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.986+0000 7fa2b37fe640 4 set_mon_vals no callback set 2024-05-15T12:32:51.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.987+0000 7fa2c34be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:51.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.987+0000 7fa2c34be640 1 -- 172.21.15.49:0/736444690 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc1477b0 con 0x7fa2bc145c00 2024-05-15T12:32:51.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.987+0000 7fa2c34be640 1 librados: init done 2024-05-15T12:32:51.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.988+0000 7fa2c34be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:51.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.988+0000 7fa2c34be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:51.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.988+0000 7fa2c34be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:51.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:51.988+0000 7fa2c34be640 1 -- 172.21.15.49:0/736444690 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc04f0e0 con 0x7fa2bc145c00 2024-05-15T12:32:51.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.007+0000 7fa2b1ffb640 1 -- 172.21.15.49:0/736444690 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fa2b4005d10 con 0x7fa2bc145c00 2024-05-15T12:32:51.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.008+0000 7fa2b1ffb640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa2a006cdd0 0x7fa2a006f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:51.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.008+0000 7fa2c125a640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa2a006cdd0 0x7fa2a006f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:51.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.008+0000 7fa2c125a640 10 monclient: get_auth_request con 0x7fa2a006cdd0 auth_method 0 2024-05-15T12:32:51.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.008+0000 7fa2c125a640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa2a006cdd0 0x7fa2a006f290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009870 tx=0x7fa2b803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:51.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.008+0000 7fa2b1ffb640 10 client.4295.objecter ms_handle_connect 0x7fa2a006cdd0 2024-05-15T12:32:52.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-15T12:32:52.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-15T12:32:52.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-15T12:32:52.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.132+0000 7f74a4379640 10 monclient: get_monmap_and_config 2024-05-15T12:32:52.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.132+0000 7f74a4379640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.132+0000 7f74a4379640 10 monclient: monmap: 2024-05-15T12:32:52.122 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.122 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.122 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:52.133819+0000 2024-05-15T12:32:52.122 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:52.133819+0000 2024-05-15T12:32:52.122 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.123 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.123 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:52.123 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:52.123 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:52.123 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:52.123 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:52.123 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.133+0000 7f74a4379640 1 Processor -- start 2024-05-15T12:32:52.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.133+0000 7f74a4379640 1 -- start start 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.133+0000 7f74a4379640 10 monclient: init 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f749c0b5410 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f749c1400a0 0x7f749c0a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 10 monclient(hunting): picked mon.noname-c con 0x7f749c1400a0 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f749c145c00 0x7f749c146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 10 monclient(hunting): picked mon.noname-b con 0x7f749c145c00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c0ab470 con 0x7f749c0b5410 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749c0b57f0 con 0x7f749c1400a0 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0a9ea0 con 0x7f749c145c00 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a4379640 10 monclient(hunting): authenticate will time out at 1530.296635s 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.134+0000 7f74a1914640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f749c1400a0 0x7f749c0a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.135+0000 7f74a1914640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f749c1400a0 0x7f749c0a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35316/0 (socket says 172.21.15.49:35316) 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.135+0000 7f74a1914640 1 -- 172.21.15.49:0/2731706752 learned_addr learned my addr 172.21.15.49:0/2731706752 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.135+0000 7f74a1914640 10 monclient(hunting): get_auth_request con 0x7f749c1400a0 auth_method 0 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.135+0000 7f74a1914640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.135+0000 7f74a1914640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.135+0000 7f74a1914640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.136+0000 7f74a1914640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.136+0000 7f74a1914640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.136+0000 7f74a1914640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.138+0000 7f74a1914640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2024-05-15T12:32:52.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.138+0000 7f74a1914640 1 -- 172.21.15.49:0/2731706752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f749c145c00 msgr2=0x7f749c146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:52.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.138+0000 7f74a1914640 1 --2- 172.21.15.49:0/2731706752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f749c145c00 0x7f749c146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.138+0000 7f74a1914640 1 -- 172.21.15.49:0/2731706752 >> v1:172.21.15.88:6789/0 conn(0x7f749c0b5410 legacy=0x7f749c13f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.138+0000 7f74a1914640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.138+0000 7f74a1914640 1 monclient: found mon.noname-c 2024-05-15T12:32:52.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.138+0000 7f74a1914640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.138+0000 7f74a1914640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:52.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.138+0000 7f74a1914640 1 -- 172.21.15.49:0/2731706752 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c0aa0c0 con 0x7f749c1400a0 2024-05-15T12:32:52.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.146+0000 7f74a1914640 1 --2- 172.21.15.49:0/2731706752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 0x7f749c0a9960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f748c009870 tx=0x7f748c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af39762d0e17f0ef server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 1 -- 172.21.15.49:0/2731706752 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c035020 con 0x7f749c1400a0 2024-05-15T12:32:52.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:52.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 10 monclient: dump: 2024-05-15T12:32:52.140 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.140 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.140 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.140 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 1 -- 172.21.15.49:0/2731706752 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748c031640 con 0x7f749c1400a0 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 1 -- 172.21.15.49:0/2731706752 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c02d040 con 0x7f749c1400a0 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a4379640 5 monclient: authenticate success, global_id 4354 2024-05-15T12:32:52.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a4379640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a4379640 4 set_mon_vals no callback set 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a1113640 10 monclient: dump: 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.142 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a4379640 10 monclient: shutdown 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a4379640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a4379640 1 -- 172.21.15.49:0/2731706752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 msgr2=0x7f749c0a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.150+0000 7f74a4379640 1 --2- 172.21.15.49:0/2731706752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 0x7f749c0a9960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f748c009870 tx=0x7f748c02b8f0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.151+0000 7f74a4379640 1 -- 172.21.15.49:0/2731706752 shutdown_connections 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.151+0000 7f74a4379640 1 --2- 172.21.15.49:0/2731706752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 0x7f749c0a9960 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.151+0000 7f74a4379640 1 --2- 172.21.15.49:0/2731706752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f749c145c00 0x7f749c146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.151+0000 7f74a4379640 1 -- 172.21.15.49:0/2731706752 >> 172.21.15.49:0/2731706752 conn(0x7f749c066f00 msgr2=0x7f749c067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.151+0000 7f74a4379640 1 -- 172.21.15.49:0/2731706752 shutdown_connections 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.151+0000 7f74a4379640 1 -- 172.21.15.49:0/2731706752 wait complete. 2024-05-15T12:32:52.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.152+0000 7f74a4379640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.152+0000 7f74a4379640 10 monclient: monmap: 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.152+0000 7f74a4379640 1 librados: starting msgr at 2024-05-15T12:32:52.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.152+0000 7f74a4379640 1 librados: starting objecter 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.152+0000 7f74a4379640 1 Processor -- start 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.152+0000 7f74a4379640 1 -- start start 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.152+0000 7f74a4379640 1 librados: setting wanted keys 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.152+0000 7f74a4379640 1 librados: calling monclient init 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.152+0000 7f74a4379640 10 monclient: init 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f749c0b5410 0x7f749c145110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 10 monclient(hunting): picked mon.noname-a con 0x7f749c0b5410 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 0x7f749c1311f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 10 monclient(hunting): picked mon.noname-b con 0x7f749c1400a0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f749c145c00 0x7f749c131730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 10 monclient(hunting): picked mon.noname-c con 0x7f749c145c00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0b4aa0 con 0x7f749c0b5410 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0b4920 con 0x7f749c1400a0 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c14cf30 con 0x7f749c145c00 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.153+0000 7f74a4379640 10 monclient(hunting): authenticate will time out at 1530.316332s 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.154+0000 7f74a1914640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 0x7f749c1311f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.154+0000 7f74a1914640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 0x7f749c1311f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35326/0 (socket says 172.21.15.49:35326) 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.154+0000 7f74a1914640 1 -- 172.21.15.49:0/2096152527 learned_addr learned my addr 172.21.15.49:0/2096152527 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.154+0000 7f74a1914640 10 monclient(hunting): get_auth_request con 0x7f749c1400a0 auth_method 0 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.154+0000 7f74a1914640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.154+0000 7f74a1914640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.154+0000 7f74a1914640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a2115640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f749c0b5410 0x7f749c145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a1914640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a1914640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a1914640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a2115640 10 monclient(hunting): get_auth_request con 0x7f749c0b5410 auth_method 0 2024-05-15T12:32:52.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a2115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a2115640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a2115640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a2115640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a2115640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.155+0000 7f74a2115640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.158+0000 7f74a1914640 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2024-05-15T12:32:52.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.158+0000 7f74a1914640 1 -- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f749c145c00 msgr2=0x7f749c131730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.158+0000 7f74a1914640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f749c145c00 0x7f749c131730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.158+0000 7f74a1914640 1 -- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f749c0b5410 msgr2=0x7f749c145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.158+0000 7f74a1914640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f749c0b5410 0x7f749c145110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.158+0000 7f74a1914640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.158+0000 7f74a1914640 1 monclient: found mon.noname-b 2024-05-15T12:32:52.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.158+0000 7f74a1914640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.158+0000 7f74a1914640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:52.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.158+0000 7f74a1914640 1 -- 172.21.15.49:0/2096152527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c139170 con 0x7f749c1400a0 2024-05-15T12:32:52.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.160+0000 7f74a1914640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 0x7f749c1311f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f748c02b930 tx=0x7f748c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.160+0000 7f7492ffd640 10 client.?.objecter ms_handle_connect 0x7f749c1400a0 2024-05-15T12:32:52.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.160+0000 7f7492ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:52.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 1 -- 172.21.15.49:0/2096152527 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c040070 con 0x7f749c1400a0 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 10 monclient: dump: 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.155 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 1 -- 172.21.15.49:0/2096152527 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748c039c10 con 0x7f749c1400a0 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f74a4379640 5 monclient: authenticate success, global_id 4357 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 1 -- 172.21.15.49:0/2096152527 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c035040 con 0x7f749c1400a0 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f7492ffd640 10 monclient: dump: 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.156 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f74a0912640 4 set_mon_vals no callback set 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f74a4379640 10 monclient: _renew_subs 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f74a4379640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f74a4379640 1 -- 172.21.15.49:0/2096152527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c131c70 con 0x7f749c1400a0 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f74a4379640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:52.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f74a4379640 10 monclient: _renew_subs 2024-05-15T12:32:52.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f74a4379640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:52.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f74a4379640 1 -- 172.21.15.49:0/2096152527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c0b5da0 con 0x7f749c1400a0 2024-05-15T12:32:52.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.164+0000 7f74a4379640 1 librados: init done 2024-05-15T12:32:52.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.165+0000 7f74a4379640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.165+0000 7f74a4379640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.165+0000 7f74a4379640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:52.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.165+0000 7f74a4379640 1 -- 172.21.15.49:0/2096152527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c04f0e0 con 0x7f749c1400a0 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.177+0000 7f867dc47640 10 monclient: get_monmap_and_config 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.177+0000 7f867dc47640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.178+0000 7f867dc47640 10 monclient: monmap: 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:52.179521+0000 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:52.179521+0000 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:52.168 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:52.169 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:52.169 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:52.169 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.178+0000 7f867dc47640 1 Processor -- start 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.178+0000 7f867dc47640 1 -- start start 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.178+0000 7f867dc47640 10 monclient: init 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86780b5410 0x7f86780a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 10 monclient(hunting): picked mon.noname-b con 0x7f86780b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f867813f990 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8678145c00 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86780ab470 con 0x7f8678145c00 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86780aa0c0 con 0x7f867813f990 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86780b57f0 con 0x7f86780b5410 2024-05-15T12:32:52.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.179+0000 7f867dc47640 10 monclient(hunting): authenticate will time out at 1530.342005s 2024-05-15T12:32:52.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.180+0000 7f8677fff640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f8678145c00 0x7f8678146070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34656/0 (socket says 172.21.15.49:34656) 2024-05-15T12:32:52.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.180+0000 7f8677fff640 1 -- 172.21.15.49:0/3711111878 learned_addr learned my addr 172.21.15.49:0/3711111878 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:52.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.183+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83841706 0 0) 0x7f86780ab470 con 0x7f8678145c00 2024-05-15T12:32:52.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.183+0000 7f86767fc640 10 monclient(hunting): _negotiate 2024-05-15T12:32:52.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.183+0000 7f86767fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.183+0000 7f86767fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.183+0000 7f86767fc640 10 monclient(hunting): my global_id is 4323 2024-05-15T12:32:52.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.183+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8660003520 con 0x7f8678145c00 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 <== mon.0 v1:172.21.15.49:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3022990774 0 0) 0x7f8660003520 con 0x7f8678145c00 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86780b5410 msgr2=0x7f86780a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 --2- 172.21.15.49:0/3711111878 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86780b5410 0x7f86780a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 >> v1:172.21.15.88:6789/0 conn(0x7f867813f990 legacy=0x7f867813fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 monclient: found mon.noname-a-legacy 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.49:6789/0 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 --> v1:172.21.15.49:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86780aa5e0 con 0x7f8678145c00 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 <== mon.0 v1:172.21.15.49:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2686122088 0 0) 0x7f86700019f0 con 0x7f8678145c00 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient: dump: 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] but i'm connected to v1:172.21.15.49:6789/0, reconnecting 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 >> v1:172.21.15.49:6789/0 conn(0x7f8678145c00 legacy=0x7f8678146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 --2- 172.21.15.49:0/3711111878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8660003a50 0x7f8660003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient(hunting): picked mon.b con 0x7f8660003a50 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 --2- 172.21.15.49:0/3711111878 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8660004800 0x7f8660004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient(hunting): picked mon.a con 0x7f8660004800 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 --2- 172.21.15.49:0/3711111878 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86780b5410 0x7f866000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient(hunting): picked mon.c con 0x7f86780b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86780b57f0 con 0x7f8660004800 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8660005620 con 0x7f8660003a50 2024-05-15T12:32:52.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866000bf30 con 0x7f86780b5410 2024-05-15T12:32:52.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.188+0000 7f86767fc640 10 monclient(hunting): _finish_auth 0 2024-05-15T12:32:52.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.189+0000 7f8676ffd640 1 --2- 172.21.15.49:0/3711111878 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8660004800 0x7f8660004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.191+0000 7f8676ffd640 10 monclient(hunting): get_auth_request con 0x7f8660004800 auth_method 0 2024-05-15T12:32:52.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.191+0000 7f8676ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.191+0000 7f8676ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.191+0000 7f8676ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T12:32:52.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.193+0000 7f8676ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.193+0000 7f8676ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.193+0000 7f8676ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T12:32:52.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 10 monclient(hunting): handle_auth_done global_id 4323 payload 293 2024-05-15T12:32:52.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 1 -- 172.21.15.49:0/3711111878 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86780b5410 msgr2=0x7f866000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:52.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 1 --2- 172.21.15.49:0/3711111878 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86780b5410 0x7f866000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 1 -- 172.21.15.49:0/3711111878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8660003a50 msgr2=0x7f8660003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:52.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 1 --2- 172.21.15.49:0/3711111878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8660003a50 0x7f8660003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 1 monclient: found mon.a 2024-05-15T12:32:52.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 1 -- 172.21.15.49:0/3711111878 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8660003520 con 0x7f8660004800 2024-05-15T12:32:52.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f8676ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f867dc47640 5 monclient: authenticate success, global_id 4323 2024-05-15T12:32:52.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.195+0000 7f867dc47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:52.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.197+0000 7f8676ffd640 1 --2- 172.21.15.49:0/3711111878 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8660004800 0x7f8660004c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f866c001650 tx=0x7f8660005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8d144c3e2c1509 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c001220 con 0x7f8660004800 2024-05-15T12:32:52.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f86767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f86767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f86767fc640 10 monclient: dump: 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866c02be50 con 0x7f8660004800 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f867dc47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f86767fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f867dc47640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f86767fc640 1 -- 172.21.15.49:0/3711111878 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c003610 con 0x7f8660004800 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f867dc47640 4 set_mon_vals no callback set 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f86767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f86767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.201+0000 7f86767fc640 10 monclient: dump: 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.193 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.202+0000 7f867dc47640 10 monclient: shutdown 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.202+0000 7f867dc47640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.202+0000 7f867dc47640 1 -- 172.21.15.49:0/3711111878 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8660004800 msgr2=0x7f8660004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.202+0000 7f867dc47640 1 --2- 172.21.15.49:0/3711111878 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8660004800 0x7f8660004c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f866c001650 tx=0x7f8660005830 comp rx=0 tx=0).stop 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.202+0000 7f8677fff640 1 -- 172.21.15.49:0/3711111878 reap_dead start 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.202+0000 7f867dc47640 1 -- 172.21.15.49:0/3711111878 shutdown_connections 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.202+0000 7f867dc47640 1 -- 172.21.15.49:0/3711111878 >> 172.21.15.49:0/3711111878 conn(0x7f8678066f00 msgr2=0x7f8678067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.202+0000 7f867dc47640 1 -- 172.21.15.49:0/3711111878 shutdown_connections 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.202+0000 7f867dc47640 1 -- 172.21.15.49:0/3711111878 wait complete. 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.203+0000 7f867dc47640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.203+0000 7f867dc47640 10 monclient: monmap: 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:52.197 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.204+0000 7f867dc47640 1 librados: starting msgr at 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.204+0000 7f867dc47640 1 librados: starting objecter 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.204+0000 7f867dc47640 1 Processor -- start 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.204+0000 7f867dc47640 1 -- start start 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.204+0000 7f867dc47640 1 librados: setting wanted keys 2024-05-15T12:32:52.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.204+0000 7f867dc47640 1 librados: calling monclient init 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.204+0000 7f867dc47640 10 monclient: init 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8660004800 0x7f86780b88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 10 monclient(hunting): picked mon.noname-b con 0x7f8660004800 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86781353a0 0x7f8678135820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 10 monclient(hunting): picked mon.noname-a con 0x7f86781353a0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86780b6160 0x7f86780b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 10 monclient(hunting): picked mon.noname-c con 0x7f86780b6160 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86780a9960 con 0x7f86781353a0 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86780a9ea0 con 0x7f8660004800 2024-05-15T12:32:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86780a9ae0 con 0x7f86780b6160 2024-05-15T12:32:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.205+0000 7f867dc47640 10 monclient(hunting): authenticate will time out at 1530.368099s 2024-05-15T12:32:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.206+0000 7f8676ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86781353a0 0x7f8678135820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.206+0000 7f8676ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86781353a0 0x7f8678135820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39138/0 (socket says 172.21.15.49:39138) 2024-05-15T12:32:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.206+0000 7f8676ffd640 1 -- 172.21.15.49:0/548861511 learned_addr learned my addr 172.21.15.49:0/548861511 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.206+0000 7f8676ffd640 10 monclient(hunting): get_auth_request con 0x7f86781353a0 auth_method 0 2024-05-15T12:32:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.206+0000 7f8676ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.206+0000 7f8676ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.206+0000 7f8676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.207+0000 7f8676ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.207+0000 7f8676ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.207+0000 7f8676ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.211+0000 7f8676ffd640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-05-15T12:32:52.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.211+0000 7f8676ffd640 1 -- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86780b6160 msgr2=0x7f86780b65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.211+0000 7f8676ffd640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86780b6160 0x7f86780b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.211+0000 7f8676ffd640 1 -- 172.21.15.49:0/548861511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8660004800 msgr2=0x7f86780b88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.211+0000 7f8676ffd640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8660004800 0x7f86780b88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.211+0000 7f8676ffd640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.211+0000 7f8676ffd640 1 monclient: found mon.noname-a 2024-05-15T12:32:52.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.211+0000 7f8676ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.212+0000 7f8676ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.212+0000 7f8676ffd640 1 -- 172.21.15.49:0/548861511 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86780b7bc0 con 0x7f86781353a0 2024-05-15T12:32:52.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.216+0000 7f8676ffd640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86781353a0 0x7f8678135820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f866c02bbe0 tx=0x7f866c000fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.216+0000 7f865ffff640 10 client.?.objecter ms_handle_connect 0x7f86781353a0 2024-05-15T12:32:52.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.216+0000 7f865ffff640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:52.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 1 -- 172.21.15.49:0/548861511 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c038040 con 0x7f86781353a0 2024-05-15T12:32:52.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:52.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 10 monclient: dump: 2024-05-15T12:32:52.211 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 1 -- 172.21.15.49:0/548861511 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866c042070 con 0x7f86781353a0 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f867dc47640 5 monclient: authenticate success, global_id 4326 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f867dc47640 10 monclient: _renew_subs 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f867dc47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f867dc47640 1 -- 172.21.15.49:0/548861511 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8678135d60 con 0x7f86781353a0 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 1 -- 172.21.15.49:0/548861511 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c007460 con 0x7f86781353a0 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f865ffff640 10 monclient: dump: 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f8675ffb640 4 set_mon_vals no callback set 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f867dc47640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f867dc47640 10 monclient: _renew_subs 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f867dc47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f867dc47640 1 -- 172.21.15.49:0/548861511 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86780b8110 con 0x7f86781353a0 2024-05-15T12:32:52.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.221+0000 7f867dc47640 1 librados: init done 2024-05-15T12:32:52.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.222+0000 7f867dc47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.222+0000 7f867dc47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.222+0000 7f867dc47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.222+0000 7f867dc47640 1 -- 172.21.15.49:0/548861511 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867804f0e0 con 0x7f86781353a0 2024-05-15T12:32:52.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.249+0000 7f74b8a13640 10 monclient: get_monmap_and_config 2024-05-15T12:32:52.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.249+0000 7f74b8a13640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.249+0000 7f74b8a13640 10 monclient: monmap: 2024-05-15T12:32:52.241 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.241 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.241 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:52.251029+0000 2024-05-15T12:32:52.241 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:52.251029+0000 2024-05-15T12:32:52.242 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.242 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.242 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:52.242 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:52.242 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:52.242 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:52.242 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:52.242 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.250+0000 7f74b8a13640 1 Processor -- start 2024-05-15T12:32:52.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.250+0000 7f74b8a13640 1 -- start start 2024-05-15T12:32:52.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.250+0000 7f74b8a13640 10 monclient: init 2024-05-15T12:32:52.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.250+0000 7f74b8a13640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.250+0000 7f74b8a13640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.250+0000 7f74b8a13640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f74b40b53e0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.250+0000 7f74b8a13640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74b4142870 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.250+0000 7f74b8a13640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f74b414b390 0x7f74b414d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.251+0000 7f74b8a13640 10 monclient(hunting): picked mon.noname-a con 0x7f74b414b390 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.251+0000 7f74b8a13640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40ab230 con 0x7f74b40b53e0 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.251+0000 7f74b8a13640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40ab510 con 0x7f74b4142870 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.251+0000 7f74b8a13640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.251+0000 7f74b8a13640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b40b57c0 con 0x7f74b414b390 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.251+0000 7f74b8a13640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.251+0000 7f74b8a13640 10 monclient(hunting): authenticate will time out at 1530.413491s 2024-05-15T12:32:52.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.254+0000 7f865ffff640 1 -- 172.21.15.49:0/548861511 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f866c0067b0 con 0x7f86781353a0 2024-05-15T12:32:52.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.254+0000 7f865ffff640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f866406ce50 0x7f866406f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.255+0000 7f86777fe640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f866406ce50 0x7f866406f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.255+0000 7f86777fe640 10 monclient: get_auth_request con 0x7f866406ce50 auth_method 0 2024-05-15T12:32:52.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.255+0000 7f865ffff640 10 client.4326.objecter ms_handle_connect 0x7f866406ce50 2024-05-15T12:32:52.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.255+0000 7f86777fe640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f866406ce50 0x7f866406f310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8668009710 tx=0x7f8668006d00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.259+0000 7f865ffff640 1 -- 172.21.15.49:0/548861511 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f866c02bdf0 con 0x7f86781353a0 2024-05-15T12:32:52.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.259+0000 7f865ffff640 10 client.4326.objecter ms_dispatch 0x7f8678000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:52.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.259+0000 7f865ffff640 3 client.4326.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:52.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.259+0000 7f865ffff640 3 client.4326.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:52.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.260+0000 7f865ffff640 20 client.4326.objecter dump_active .. 0 homeless 2024-05-15T12:32:52.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.301+0000 7fa2b1ffb640 1 -- 172.21.15.49:0/736444690 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa2b4057d10 con 0x7fa2bc145c00 2024-05-15T12:32:52.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.301+0000 7fa2b1ffb640 10 client.4295.objecter ms_dispatch 0x7fa2bc000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:52.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.301+0000 7fa2b1ffb640 3 client.4295.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:52.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.301+0000 7fa2b1ffb640 3 client.4295.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:52.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.301+0000 7fa2b1ffb640 20 client.4295.objecter dump_active .. 0 homeless 2024-05-15T12:32:52.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.301+0000 7fa2b1ffb640 1 -- 172.21.15.49:0/736444690 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa2b4022070 con 0x7fa2bc145c00 2024-05-15T12:32:52.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.301+0000 7fa2b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.301+0000 7fa2b1ffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:52.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.413+0000 7f7492ffd640 1 -- 172.21.15.49:0/2096152527 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f748c04b060 con 0x7f749c1400a0 2024-05-15T12:32:52.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.414+0000 7f7492ffd640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f748006cdd0 0x7f748006f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.414+0000 7f74a2115640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f748006cdd0 0x7f748006f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.414+0000 7f74a2115640 10 monclient: get_auth_request con 0x7f748006cdd0 auth_method 0 2024-05-15T12:32:52.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.415+0000 7f7492ffd640 10 client.4357.objecter ms_handle_connect 0x7f748006cdd0 2024-05-15T12:32:52.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.415+0000 7f74a2115640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f748006cdd0 0x7f748006f290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7498001780 tx=0x7f7498009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.417+0000 7f7492ffd640 1 -- 172.21.15.49:0/2096152527 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f748c0315f0 con 0x7f749c1400a0 2024-05-15T12:32:52.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.417+0000 7f7492ffd640 10 client.4357.objecter ms_dispatch 0x7f749c000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:52.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.417+0000 7f7492ffd640 3 client.4357.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:52.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.417+0000 7f7492ffd640 3 client.4357.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:52.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.417+0000 7f7492ffd640 20 client.4357.objecter dump_active .. 0 homeless 2024-05-15T12:32:52.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.427+0000 7f7492ffd640 1 -- 172.21.15.49:0/2096152527 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f748c07af00 con 0x7f749c1400a0 2024-05-15T12:32:52.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.427+0000 7f7492ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.427+0000 7f7492ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:52.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.432+0000 7f74abfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f74b4142870 0x7f74b4142cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:36962/0 (socket says 172.21.15.49:36962) 2024-05-15T12:32:52.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.432+0000 7f74abfff640 1 -- 172.21.15.49:0/2173667027 learned_addr learned my addr 172.21.15.49:0/2173667027 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:52.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.490+0000 7fa2c34be640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:32:52.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.490+0000 7fa2c34be640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:32:52.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.490+0000 7fa2c34be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:52.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.490+0000 7fa2c34be640 1 -- 172.21.15.49:0/736444690 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa2bc146ad0 con 0x7fa2bc145c00 2024-05-15T12:32:52.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.495+0000 7fa2b1ffb640 1 -- 172.21.15.49:0/736444690 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa2b4010040 con 0x7fa2bc145c00 2024-05-15T12:32:52.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.495+0000 7fa2b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-15T12:32:52.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.495+0000 7fa2b1ffb640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:52.486 INFO:teuthology.orchestra.run.smithi049.stdout:55834574854 2024-05-15T12:32:52.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 10 librados: watch_flush enter 2024-05-15T12:32:52.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 10 librados: watch_flush exit 2024-05-15T12:32:52.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 20 client.4295.objecter shutdown clearing up homeless session... 2024-05-15T12:32:52.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 10 client.4295.objecter successfully canceled tick 2024-05-15T12:32:52.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 -- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa2a006cdd0 msgr2=0x7fa2a006f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa2a006cdd0 0x7fa2a006f290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009870 tx=0x7fa2b803c040 comp rx=0 tx=0).stop 2024-05-15T12:32:52.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 10 monclient: shutdown 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 -- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc145c00 msgr2=0x7fa2bc142d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc145c00 0x7fa2bc142d10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400ec10 tx=0x7fa2b400c490 comp rx=0 tx=0).stop 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 -- 172.21.15.49:0/736444690 shutdown_connections 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2bc145c00 0x7fa2bc142d10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa2a006cdd0 0x7fa2a006f290 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2bc13f990 0x7fa2bc147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 --2- 172.21.15.49:0/736444690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2bc0b5410 0x7fa2bc1484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 -- 172.21.15.49:0/736444690 >> 172.21.15.49:0/736444690 conn(0x7fa2bc066f00 msgr2=0x7fa2bc0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 -- 172.21.15.49:0/736444690 shutdown_connections 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 -- 172.21.15.49:0/736444690 wait complete. 2024-05-15T12:32:52.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.498+0000 7fa2c34be640 1 librados: shutdown 2024-05-15T12:32:52.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-15T12:32:52.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-05-15T12:32:52.499 DEBUG:teuthology.parallel:result is None 2024-05-15T12:32:52.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.528+0000 7f865ffff640 1 -- 172.21.15.49:0/548861511 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f866c04c020 con 0x7f86781353a0 2024-05-15T12:32:52.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.528+0000 7f865ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.528+0000 7f865ffff640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:52.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.529+0000 7f74b2d9d640 1 --2- 172.21.15.49:0/2173667027 >> v2:172.21.15.49:3300/0 conn(0x7f74b414b390 0x7f74b414d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.534+0000 7f74b2d9d640 10 monclient(hunting): get_auth_request con 0x7f74b414b390 auth_method 0 2024-05-15T12:32:52.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.534+0000 7f74b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.534+0000 7f74b2d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.534+0000 7f74b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.538+0000 7f74b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.538+0000 7f74b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.538+0000 7f74b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.545+0000 7f74b2d9d640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2024-05-15T12:32:52.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.545+0000 7f74b2d9d640 1 -- 172.21.15.49:0/2173667027 >> v1:172.21.15.88:6789/0 conn(0x7f74b4142870 legacy=0x7f74b4142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.545+0000 7f74b2d9d640 1 -- 172.21.15.49:0/2173667027 >> v1:172.21.15.49:6789/0 conn(0x7f74b40b53e0 legacy=0x7f74b40a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.545+0000 7f74b2d9d640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.545+0000 7f74b2d9d640 1 monclient: found mon.noname-a 2024-05-15T12:32:52.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.545+0000 7f74b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.545+0000 7f74b2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.545+0000 7f74b2d9d640 1 -- 172.21.15.49:0/2173667027 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b41433c0 con 0x7f74b414b390 2024-05-15T12:32:52.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.560+0000 7f74b2d9d640 1 --2- 172.21.15.49:0/2173667027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74b414b390 0x7f74b414d780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009d90 tx=0x7f74ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd517a4c0b25a110 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-15T12:32:52.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.632+0000 7f8d2cdf8640 10 monclient: get_monmap_and_config 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.632+0000 7f8d2cdf8640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.632+0000 7f8d2cdf8640 10 monclient: monmap: 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:52.634380+0000 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:52.634380+0000 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:52.623 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.633+0000 7f8d2cdf8640 1 Processor -- start 2024-05-15T12:32:52.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.633+0000 7f8d2cdf8640 1 -- start start 2024-05-15T12:32:52.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.633+0000 7f8d2cdf8640 10 monclient: init 2024-05-15T12:32:52.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.634+0000 7f8d2cdf8640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.634+0000 7f8d2cdf8640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-05-15T12:32:52.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.634+0000 7f8d2cdf8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d280b5410 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:52.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.634+0000 7f8d2cdf8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d281400a0 0x7f8d280a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.634+0000 7f8d2cdf8640 10 monclient(hunting): picked mon.noname-b con 0x7f8d281400a0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:52.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.634+0000 7f8d2cdf8640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8d28145c00 0x7f8d28146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.635+0000 7f8d2cdf8640 10 monclient(hunting): picked mon.noname-a con 0x7f8d28145c00 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:52.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.635+0000 7f8d2cdf8640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d280ab470 con 0x7f8d280b5410 2024-05-15T12:32:52.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.635+0000 7f8d2cdf8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.635+0000 7f8d2cdf8640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d280b57f0 con 0x7f8d28145c00 2024-05-15T12:32:52.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.635+0000 7f8d2cdf8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.635+0000 7f8d2cdf8640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d280a9ea0 con 0x7f8d281400a0 2024-05-15T12:32:52.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.635+0000 7f8d2cdf8640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.635+0000 7f8d2cdf8640 10 monclient(hunting): authenticate will time out at 1530.797493s 2024-05-15T12:32:52.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-15T12:32:52.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 1 -- 172.21.15.49:0/2173667027 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac039070 con 0x7f74b414b390 2024-05-15T12:32:52.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:52.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 10 monclient: dump: 2024-05-15T12:32:52.660 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 1 -- 172.21.15.49:0/2173667027 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac02cc60 con 0x7f74b414b390 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 1 -- 172.21.15.49:0/2173667027 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac034660 con 0x7f74b414b390 2024-05-15T12:32:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b1d9b640 10 monclient: dump: 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b8a13640 5 monclient: authenticate success, global_id 4329 2024-05-15T12:32:52.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b8a13640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:52.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f74b8a13640 4 set_mon_vals no callback set 2024-05-15T12:32:52.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.670+0000 7f8d25d9b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d281400a0 0x7f8d280a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d26d9d640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8d28145c00 0x7f8d28146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f74b8a13640 10 monclient: shutdown 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f74b8a13640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f74b8a13640 1 -- 172.21.15.49:0/2173667027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74b414b390 msgr2=0x7f74b414d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f74b8a13640 1 --2- 172.21.15.49:0/2173667027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74b414b390 0x7f74b414d780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009d90 tx=0x7f74ac02beb0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d25d9b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d281400a0 0x7f8d280a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46314/0 (socket says 172.21.15.49:46314) 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d25d9b640 1 -- 172.21.15.49:0/1891152998 learned_addr learned my addr 172.21.15.49:0/1891152998 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d25d9b640 10 monclient(hunting): get_auth_request con 0x7f8d281400a0 auth_method 0 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d25d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d26d9d640 10 monclient(hunting): get_auth_request con 0x7f8d28145c00 auth_method 0 2024-05-15T12:32:52.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d26d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f8d26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f74b8a13640 1 -- 172.21.15.49:0/2173667027 shutdown_connections 2024-05-15T12:32:52.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f74b8a13640 1 --2- 172.21.15.49:0/2173667027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74b414b390 0x7f74b414d780 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f74b8a13640 1 -- 172.21.15.49:0/2173667027 >> 172.21.15.49:0/2173667027 conn(0x7f74b4066ed0 msgr2=0x7f74b40672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:52.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f74b8a13640 1 -- 172.21.15.49:0/2173667027 shutdown_connections 2024-05-15T12:32:52.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.671+0000 7f74b8a13640 1 -- 172.21.15.49:0/2173667027 wait complete. 2024-05-15T12:32:52.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.672+0000 7f8d26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.672+0000 7f8d26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.672+0000 7f8d26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.672+0000 7f8d25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.672+0000 7f8d25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.672+0000 7f8d25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.672+0000 7f74a4379640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:32:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.673+0000 7f74a4379640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:32:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.673+0000 7f74a4379640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.673+0000 7f74a4379640 1 -- 172.21.15.49:0/2096152527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f749c139340 con 0x7f749c1400a0 2024-05-15T12:32:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.673+0000 7f74b8a13640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.673+0000 7f74b8a13640 10 monclient: monmap: 2024-05-15T12:32:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.673+0000 7f74b8a13640 1 librados: starting msgr at 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f74b8a13640 1 librados: starting objecter 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f74b8a13640 1 Processor -- start 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f74b8a13640 1 -- start start 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f74b8a13640 1 librados: setting wanted keys 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f74b8a13640 1 librados: calling monclient init 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f74b8a13640 10 monclient: init 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f8d26d9d640 10 monclient(hunting): handle_auth_done global_id 4332 payload 274 2024-05-15T12:32:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f8d26d9d640 1 -- 172.21.15.49:0/1891152998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d281400a0 msgr2=0x7f8d280a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f8d26d9d640 1 --2- 172.21.15.49:0/1891152998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d281400a0 0x7f8d280a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f8d26d9d640 1 -- 172.21.15.49:0/1891152998 >> v1:172.21.15.49:6789/0 conn(0x7f8d280b5410 legacy=0x7f8d2813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f8d26d9d640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f8d26d9d640 1 monclient: found mon.noname-a 2024-05-15T12:32:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f8d26d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f8d26d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.674+0000 7f8d26d9d640 1 -- 172.21.15.49:0/1891152998 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d280aa0c0 con 0x7f8d28145c00 2024-05-15T12:32:52.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.675+0000 7f74b8a13640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.675+0000 7f74b8a13640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.675+0000 7f74b8a13640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b40b53e0 0x7f74b41db710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.675+0000 7f74b8a13640 10 monclient(hunting): picked mon.noname-b con 0x7f74b40b53e0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.675+0000 7f74b8a13640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74b4142870 0x7f74b4132300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.675+0000 7f74b8a13640 10 monclient(hunting): picked mon.noname-c con 0x7f74b4142870 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.675+0000 7f74b8a13640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74b414b390 0x7f74b4132840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f74b8a13640 10 monclient(hunting): picked mon.noname-a con 0x7f74b414b390 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f74b8a13640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f74b8a13640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b414ded0 con 0x7f74b414b390 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f74b8a13640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f74b8a13640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b40aa230 con 0x7f74b40b53e0 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f74b8a13640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f74b8a13640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b414e050 con 0x7f74b4142870 2024-05-15T12:32:52.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f7492ffd640 1 -- 172.21.15.49:0/2096152527 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f748c07a800 con 0x7f749c1400a0 2024-05-15T12:32:52.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f74b8a13640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f74b8a13640 10 monclient(hunting): authenticate will time out at 1530.839073s 2024-05-15T12:32:52.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f7492ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-15T12:32:52.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.676+0000 7f7492ffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:52.674 INFO:teuthology.orchestra.run.smithi049.stdout:60129542153 2024-05-15T12:32:52.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.677+0000 7f74b259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b40b53e0 0x7f74b41db710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.678+0000 7f74b2d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74b414b390 0x7f74b4132840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.678+0000 7f74b259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b40b53e0 0x7f74b41db710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35360/0 (socket says 172.21.15.49:35360) 2024-05-15T12:32:52.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.678+0000 7f74b259c640 1 -- 172.21.15.49:0/1605078804 learned_addr learned my addr 172.21.15.49:0/1605078804 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:52.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.678+0000 7f74b259c640 10 monclient(hunting): get_auth_request con 0x7f74b40b53e0 auth_method 0 2024-05-15T12:32:52.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.678+0000 7f74b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.678+0000 7f74b259c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.678+0000 7f74b259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 10 librados: watch_flush enter 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 10 librados: watch_flush exit 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 20 client.4357.objecter shutdown clearing up homeless session... 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 10 client.4357.objecter successfully canceled tick 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 -- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f748006cdd0 msgr2=0x7f748006f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f748006cdd0 0x7f748006f290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7498001780 tx=0x7f7498009040 comp rx=0 tx=0).stop 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 10 monclient: shutdown 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 -- 172.21.15.49:0/2096152527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 msgr2=0x7f749c1311f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 0x7f749c1311f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f748c02b930 tx=0x7f748c0046c0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 -- 172.21.15.49:0/2096152527 shutdown_connections 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f749c145c00 0x7f749c131730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f748006cdd0 0x7f748006f290 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f749c1400a0 0x7f749c1311f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 --2- 172.21.15.49:0/2096152527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f749c0b5410 0x7f749c145110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 -- 172.21.15.49:0/2096152527 >> 172.21.15.49:0/2096152527 conn(0x7f749c066f00 msgr2=0x7f749c13a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:52.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f8d26d9d640 1 --2- 172.21.15.49:0/1891152998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d28145c00 0x7f8d28146070 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009dc0 tx=0x7f8d2002bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875de8022ee06cf9 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 -- 172.21.15.49:0/2096152527 shutdown_connections 2024-05-15T12:32:52.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.679+0000 7f74a4379640 1 -- 172.21.15.49:0/2096152527 wait complete. 2024-05-15T12:32:52.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.680+0000 7f74a4379640 1 librados: shutdown 2024-05-15T12:32:52.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.680+0000 7f74b2d9d640 10 monclient(hunting): get_auth_request con 0x7f74b414b390 auth_method 0 2024-05-15T12:32:52.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.680+0000 7f74b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.680+0000 7f74b2d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.680+0000 7f74b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.682+0000 7f74b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.682+0000 7f74b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.682+0000 7f74b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.683+0000 7f74b259c640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.683+0000 7f74b259c640 1 -- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74b4142870 msgr2=0x7f74b4132300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.683+0000 7f74b259c640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74b4142870 0x7f74b4132300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.683+0000 7f74b259c640 1 -- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74b414b390 msgr2=0x7f74b4132840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.683+0000 7f74b259c640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74b414b390 0x7f74b4132840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.683+0000 7f74b259c640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.683+0000 7f74b259c640 1 monclient: found mon.noname-b 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.683+0000 7f74b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.683+0000 7f74b259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.683+0000 7f74b259c640 1 -- 172.21.15.49:0/1605078804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b40b8310 con 0x7f74b40b53e0 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.686+0000 7f74aaffd640 10 client.?.objecter ms_handle_connect 0x7f74b40b53e0 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.686+0000 7f74aaffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:52.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.686+0000 7f74b259c640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b40b53e0 0x7f74b41db710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f74a400c430 tx=0x7f74a400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 1 -- 172.21.15.49:0/1605078804 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a40052d0 con 0x7f74b40b53e0 2024-05-15T12:32:52.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:52.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 10 monclient: dump: 2024-05-15T12:32:52.681 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.681 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.681 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 1 -- 172.21.15.49:0/1605078804 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a4005470 con 0x7f74b40b53e0 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 1 -- 172.21.15.49:0/1605078804 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a40052d0 con 0x7f74b40b53e0 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74aaffd640 10 monclient: dump: 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f74b159a640 4 set_mon_vals no callback set 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f74b8a13640 5 monclient: authenticate success, global_id 4360 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f74b8a13640 10 monclient: _renew_subs 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f74b8a13640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:52.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f74b8a13640 1 -- 172.21.15.49:0/1605078804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b4132d80 con 0x7f74b40b53e0 2024-05-15T12:32:52.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f74b8a13640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:52.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f74b8a13640 10 monclient: _renew_subs 2024-05-15T12:32:52.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f74b8a13640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:52.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f74b8a13640 1 -- 172.21.15.49:0/1605078804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b40b8860 con 0x7f74b40b53e0 2024-05-15T12:32:52.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f74b8a13640 1 librados: init done 2024-05-15T12:32:52.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f8d2559a640 1 -- 172.21.15.49:0/1891152998 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20039070 con 0x7f8d28145c00 2024-05-15T12:32:52.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f8d2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f8d2559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f8d2559a640 10 monclient: dump: 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f8d2559a640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f8d2559a640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.690+0000 7f8d2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f8d2559a640 1 -- 172.21.15.49:0/1891152998 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2002cb10 con 0x7f8d28145c00 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f8d2559a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f8d2559a640 1 -- 172.21.15.49:0/1891152998 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20034440 con 0x7f8d28145c00 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f8d2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f8d2559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f8d2559a640 10 monclient: dump: 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.686 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f8d2cdf8640 5 monclient: authenticate success, global_id 4332 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f8d2cdf8640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.691+0000 7f8d2cdf8640 4 set_mon_vals no callback set 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.692+0000 7f8d2cdf8640 10 monclient: shutdown 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.692+0000 7f8d2cdf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.692+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1891152998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d28145c00 msgr2=0x7f8d28146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.692+0000 7f8d2cdf8640 1 --2- 172.21.15.49:0/1891152998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d28145c00 0x7f8d28146070 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009dc0 tx=0x7f8d2002bd60 comp rx=0 tx=0).stop 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.692+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1891152998 shutdown_connections 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.692+0000 7f8d2cdf8640 1 --2- 172.21.15.49:0/1891152998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d28145c00 0x7f8d28146070 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.692+0000 7f8d2cdf8640 1 --2- 172.21.15.49:0/1891152998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d281400a0 0x7f8d280a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.693+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1891152998 >> 172.21.15.49:0/1891152998 conn(0x7f8d28066f00 msgr2=0x7f8d28067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:52.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.693+0000 7f74a8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.693+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1891152998 shutdown_connections 2024-05-15T12:32:52.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.693+0000 7f74a8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.693+0000 7f74a8ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:52.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.693+0000 7f74a8ff9640 1 -- 172.21.15.49:0/1605078804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b40b7700 con 0x7f74b40b53e0 2024-05-15T12:32:52.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.693+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1891152998 wait complete. 2024-05-15T12:32:52.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.694+0000 7f8d2cdf8640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.694+0000 7f8d2cdf8640 10 monclient: monmap: 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.694+0000 7f8d2cdf8640 1 librados: starting msgr at 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.694+0000 7f8d2cdf8640 1 librados: starting objecter 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.695+0000 7f8d2cdf8640 1 Processor -- start 2024-05-15T12:32:52.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.695+0000 7f8d2cdf8640 1 -- start start 2024-05-15T12:32:52.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.695+0000 7f8d2cdf8640 1 librados: setting wanted keys 2024-05-15T12:32:52.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.695+0000 7f8d2cdf8640 1 librados: calling monclient init 2024-05-15T12:32:52.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.695+0000 7f8d2cdf8640 10 monclient: init 2024-05-15T12:32:52.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.6 2024-05-15T12:32:52.692 DEBUG:teuthology.parallel:result is None 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d280b5410 0x7f8d28134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 10 monclient(hunting): picked mon.noname-c con 0x7f8d280b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d281400a0 0x7f8d28142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 10 monclient(hunting): picked mon.noname-b con 0x7f8d281400a0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d28145c00 0x7f8d28141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 10 monclient(hunting): picked mon.noname-a con 0x7f8d28145c00 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d280b4aa0 con 0x7f8d28145c00 2024-05-15T12:32:52.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d280b4920 con 0x7f8d281400a0 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2814cf30 con 0x7f8d280b5410 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.696+0000 7f8d2cdf8640 10 monclient(hunting): authenticate will time out at 1530.858778s 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.698+0000 7f8d2659c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d280b5410 0x7f8d28134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.698+0000 7f8d2659c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d280b5410 0x7f8d28134c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46324/0 (socket says 172.21.15.49:46324) 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.698+0000 7f8d2659c640 1 -- 172.21.15.49:0/1708148274 learned_addr learned my addr 172.21.15.49:0/1708148274 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.698+0000 7f8d2659c640 10 monclient(hunting): get_auth_request con 0x7f8d280b5410 auth_method 0 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.698+0000 7f8d2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.698+0000 7f8d2659c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.698+0000 7f8d2659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.699+0000 7f8d2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.699+0000 7f8d2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.699+0000 7f8d2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.704+0000 7f8d2659c640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.704+0000 7f8d2659c640 1 -- 172.21.15.49:0/1708148274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d281400a0 msgr2=0x7f8d28142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.704+0000 7f8d2659c640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d281400a0 0x7f8d28142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.704+0000 7f8d2659c640 1 -- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d28145c00 msgr2=0x7f8d28141160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.704+0000 7f8d2659c640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d28145c00 0x7f8d28141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.704+0000 7f8d2659c640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.704+0000 7f8d2659c640 1 monclient: found mon.noname-c 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.704+0000 7f8d2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.704+0000 7f8d2659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.704+0000 7f8d2659c640 1 -- 172.21.15.49:0/1708148274 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d28135750 con 0x7f8d280b5410 2024-05-15T12:32:52.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.707+0000 7f8d2659c640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d280b5410 0x7f8d28134c80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800c410 tx=0x7f8d1800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.707+0000 7f8d16ffd640 10 client.?.objecter ms_handle_connect 0x7f8d280b5410 2024-05-15T12:32:52.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.707+0000 7f8d16ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 1 -- 172.21.15.49:0/1708148274 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18017070 con 0x7f8d280b5410 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 10 monclient: dump: 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.702 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 1 -- 172.21.15.49:0/1708148274 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d180035f0 con 0x7f8d280b5410 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 1 -- 172.21.15.49:0/1708148274 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d180039e0 con 0x7f8d280b5410 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d16ffd640 10 monclient: dump: 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.703 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d2cdf8640 5 monclient: authenticate success, global_id 4301 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d2cdf8640 10 monclient: _renew_subs 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d2cdf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1708148274 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d281391d0 con 0x7f8d280b5410 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.711+0000 7f8d24d99640 4 set_mon_vals no callback set 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.712+0000 7f8d2cdf8640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.712+0000 7f8d2cdf8640 10 monclient: _renew_subs 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.712+0000 7f8d2cdf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.712+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1708148274 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d28140920 con 0x7f8d280b5410 2024-05-15T12:32:52.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.712+0000 7f8d2cdf8640 1 librados: init done 2024-05-15T12:32:52.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.713+0000 7f8d2cdf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.713+0000 7f8d2cdf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.713+0000 7f8d2cdf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:52.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.713+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1708148274 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d281393a0 con 0x7f8d280b5410 2024-05-15T12:32:52.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.723+0000 7fd43e59f640 10 monclient: get_monmap_and_config 2024-05-15T12:32:52.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.723+0000 7fd43e59f640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.723+0000 7fd43e59f640 10 monclient: monmap: 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:52.724607+0000 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:52.724607+0000 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:52.713 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.724+0000 7fd43e59f640 1 Processor -- start 2024-05-15T12:32:52.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.724+0000 7fd43e59f640 1 -- start start 2024-05-15T12:32:52.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.724+0000 7fd43e59f640 10 monclient: init 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.724+0000 7fd43e59f640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.724+0000 7fd43e59f640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.724+0000 7fd43e59f640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd4380726a0 0x7fd43813f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.724+0000 7fd43e59f640 10 monclient(hunting): picked mon.noname-a con 0x7fd4380726a0 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.724+0000 7fd43e59f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd438072a80 0x7fd43806fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.724+0000 7fd43e59f640 10 monclient(hunting): picked mon.noname-c con 0x7fd438072a80 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.725+0000 7fd43e59f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4380ae5e0 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.725+0000 7fd43e59f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd43813f840 con 0x7fd4380ae5e0 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.725+0000 7fd43e59f640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.725+0000 7fd43e59f640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd43806f720 con 0x7fd4380726a0 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.725+0000 7fd43e59f640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.725+0000 7fd43e59f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4380700f0 con 0x7fd438072a80 2024-05-15T12:32:52.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.725+0000 7fd43e59f640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.725+0000 7fd43e59f640 10 monclient(hunting): authenticate will time out at 1530.887614s 2024-05-15T12:32:52.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.729+0000 7fd43d59d640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd4380726a0 0x7fd43813f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.729+0000 7fd43d59d640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd4380726a0 0x7fd43813f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39188/0 (socket says 172.21.15.49:39188) 2024-05-15T12:32:52.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.729+0000 7fd43d59d640 1 -- 172.21.15.49:0/400297300 learned_addr learned my addr 172.21.15.49:0/400297300 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:52.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.729+0000 7fd43d59d640 10 monclient(hunting): get_auth_request con 0x7fd4380726a0 auth_method 0 2024-05-15T12:32:52.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.729+0000 7fd43d59d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.729+0000 7fd43d59d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.729+0000 7fd43d59d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.730+0000 7fd43d59d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.730+0000 7fd43d59d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.730+0000 7fd43d59d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.731+0000 7fd43d59d640 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2024-05-15T12:32:52.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.731+0000 7fd43d59d640 1 -- 172.21.15.49:0/400297300 >> v2:172.21.15.88:3300/0 conn(0x7fd438072a80 msgr2=0x7fd43806fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:52.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.731+0000 7fd43d59d640 1 --2- 172.21.15.49:0/400297300 >> v2:172.21.15.88:3300/0 conn(0x7fd438072a80 0x7fd43806fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.731+0000 7fd43d59d640 1 -- 172.21.15.49:0/400297300 >> v1:172.21.15.88:6789/0 conn(0x7fd4380ae5e0 legacy=0x7fd438142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:52.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.731+0000 7fd43d59d640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.731+0000 7fd43d59d640 1 monclient: found mon.noname-a 2024-05-15T12:32:52.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.731+0000 7fd43d59d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.731+0000 7fd43d59d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.731+0000 7fd43d59d640 1 -- 172.21.15.49:0/400297300 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd438070dc0 con 0x7fd4380726a0 2024-05-15T12:32:52.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.733+0000 7fd43d59d640 1 --2- 172.21.15.49:0/400297300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 0x7fd43813f300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd438071820 tx=0x7fd43402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f215648eed45bf server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 1 -- 172.21.15.49:0/400297300 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434034070 con 0x7fd4380726a0 2024-05-15T12:32:52.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 10 monclient: dump: 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 1 -- 172.21.15.49:0/400297300 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd434030860 con 0x7fd4380726a0 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd43e59f640 5 monclient: authenticate success, global_id 4338 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd43e59f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd43e59f640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 1 -- 172.21.15.49:0/400297300 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434030c30 con 0x7fd4380726a0 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd43e59f640 4 set_mon_vals no callback set 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.737+0000 7fd42ffff640 10 monclient: dump: 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.729 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.730 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.730 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.730 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.730 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.730 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.738+0000 7fd43e59f640 10 monclient: shutdown 2024-05-15T12:32:52.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.738+0000 7fd43e59f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.738+0000 7fd43e59f640 1 -- 172.21.15.49:0/400297300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 msgr2=0x7fd43813f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.738+0000 7fd43e59f640 1 --2- 172.21.15.49:0/400297300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 0x7fd43813f300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd438071820 tx=0x7fd43402b9a0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.738+0000 7fd43e59f640 1 -- 172.21.15.49:0/400297300 shutdown_connections 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.738+0000 7fd43e59f640 1 --2- 172.21.15.49:0/400297300 >> v2:172.21.15.88:3300/0 conn(0x7fd438072a80 0x7fd43806fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.738+0000 7fd43e59f640 1 --2- 172.21.15.49:0/400297300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 0x7fd43813f300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.738+0000 7fd43e59f640 1 -- 172.21.15.49:0/400297300 >> 172.21.15.49:0/400297300 conn(0x7fd43806af80 msgr2=0x7fd43806b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.739+0000 7fd43e59f640 1 -- 172.21.15.49:0/400297300 shutdown_connections 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.739+0000 7fd43e59f640 1 -- 172.21.15.49:0/400297300 wait complete. 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.740+0000 7fd43e59f640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.740+0000 7fd43e59f640 10 monclient: monmap: 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.740+0000 7fd43e59f640 1 librados: starting msgr at 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.740+0000 7fd43e59f640 1 librados: starting objecter 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.740+0000 7fd43e59f640 1 Processor -- start 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.740+0000 7fd43e59f640 1 -- start start 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.740+0000 7fd43e59f640 1 librados: setting wanted keys 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.740+0000 7fd43e59f640 1 librados: calling monclient init 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.740+0000 7fd43e59f640 10 monclient: init 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.741+0000 7fd43e59f640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.741+0000 7fd43e59f640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.741+0000 7fd43e59f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 0x7fd4380ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.741+0000 7fd43e59f640 10 monclient(hunting): picked mon.noname-a con 0x7fd4380726a0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.741+0000 7fd43e59f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd438072a80 0x7fd4381db490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.741+0000 7fd43e59f640 10 monclient(hunting): picked mon.noname-b con 0x7fd438072a80 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43e59f640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4380ae5e0 0x7fd4381db9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43e59f640 10 monclient(hunting): picked mon.noname-c con 0x7fd4380ae5e0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43e59f640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43e59f640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd438070830 con 0x7fd4380726a0 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43e59f640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43e59f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd438071f30 con 0x7fd438072a80 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43e59f640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43d59d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 0x7fd4380ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43e59f640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4380aea90 con 0x7fd4380ae5e0 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43e59f640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43e59f640 10 monclient(hunting): authenticate will time out at 1530.905157s 2024-05-15T12:32:52.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43d59d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 0x7fd4380ad8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39200/0 (socket says 172.21.15.49:39200) 2024-05-15T12:32:52.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43d59d640 1 -- 172.21.15.49:0/3831644811 learned_addr learned my addr 172.21.15.49:0/3831644811 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:52.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43d59d640 10 monclient(hunting): get_auth_request con 0x7fd4380726a0 auth_method 0 2024-05-15T12:32:52.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43d59d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:52.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43d59d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:52.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.742+0000 7fd43d59d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:52.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.743+0000 7fd43d59d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:52.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.743+0000 7fd43d59d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:52.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.743+0000 7fd43d59d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:52.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.748+0000 7fd43d59d640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-05-15T12:32:52.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.748+0000 7fd43d59d640 1 -- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4380ae5e0 msgr2=0x7fd4381db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.748+0000 7fd43d59d640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4380ae5e0 0x7fd4381db9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.748+0000 7fd43d59d640 1 -- 172.21.15.49:0/3831644811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd438072a80 msgr2=0x7fd4381db490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:52.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.748+0000 7fd43d59d640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd438072a80 0x7fd4381db490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:52.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.748+0000 7fd43d59d640 10 monclient: _finish_hunting 0 2024-05-15T12:32:52.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.748+0000 7fd43d59d640 1 monclient: found mon.noname-a 2024-05-15T12:32:52.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.748+0000 7fd43d59d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:52.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.748+0000 7fd43d59d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.748+0000 7fd43d59d640 1 -- 172.21.15.49:0/3831644811 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43813ef70 con 0x7fd4380726a0 2024-05-15T12:32:52.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.751+0000 7fd43d59d640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 0x7fd4380ad8a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd43402b9e0 tx=0x7fd434004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.751+0000 7fd42dffb640 10 client.?.objecter ms_handle_connect 0x7fd4380726a0 2024-05-15T12:32:52.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.752+0000 7fd42dffb640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 1 -- 172.21.15.49:0/3831644811 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434048070 con 0x7fd4380726a0 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 10 monclient: dump: 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 10 monclient: _finish_auth 0 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 10 monclient: _check_auth_tickets 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 1 -- 172.21.15.49:0/3831644811 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd434030660 con 0x7fd4380726a0 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 1 -- 172.21.15.49:0/3831644811 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434034040 con 0x7fd4380726a0 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42dffb640 10 monclient: dump: 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:52.748 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.749 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:52.749 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:52.749 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:52.749 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.757+0000 7fd43e59f640 5 monclient: authenticate success, global_id 4341 2024-05-15T12:32:52.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.757+0000 7fd43e59f640 10 monclient: _renew_subs 2024-05-15T12:32:52.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.757+0000 7fd43e59f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.757+0000 7fd43e59f640 1 -- 172.21.15.49:0/3831644811 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4380adde0 con 0x7fd4380726a0 2024-05-15T12:32:52.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.756+0000 7fd42f7fe640 4 set_mon_vals no callback set 2024-05-15T12:32:52.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.757+0000 7fd43e59f640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:52.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.757+0000 7fd43e59f640 10 monclient: _renew_subs 2024-05-15T12:32:52.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.757+0000 7fd43e59f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.757+0000 7fd43e59f640 1 -- 172.21.15.49:0/3831644811 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd438149760 con 0x7fd4380726a0 2024-05-15T12:32:52.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.757+0000 7fd43e59f640 1 librados: init done 2024-05-15T12:32:52.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.759+0000 7fd43e59f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.759+0000 7fd43e59f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.759+0000 7fd43e59f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.759+0000 7fd43e59f640 1 -- 172.21.15.49:0/3831644811 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43804f0e0 con 0x7fd4380726a0 2024-05-15T12:32:52.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.775+0000 7fa580290640 10 monclient: get_monmap_and_config 2024-05-15T12:32:52.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.775+0000 7fa580290640 10 monclient: build_initial_monmap 2024-05-15T12:32:52.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.775+0000 7fa580290640 10 monclient: monmap: 2024-05-15T12:32:52.765 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:52.765 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:52.765 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:52.776898+0000 2024-05-15T12:32:52.765 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:52.776898+0000 2024-05-15T12:32:52.765 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:52.765 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:52.766 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:52.766 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:52.766 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:52.766 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:52.766 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:52.766 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:52.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.775+0000 7fa580290640 1 Processor -- start 2024-05-15T12:32:52.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.776+0000 7fa580290640 1 -- start start 2024-05-15T12:32:52.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.776+0000 7fa580290640 10 monclient: init 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.776+0000 7fa580290640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.776+0000 7fa580290640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.776+0000 7fa580290640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa5780b5410 0x7fa5780a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 10 monclient(hunting): picked mon.noname-a con 0x7fa5780b5410 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa57813f990 0x7fa57813fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 10 monclient(hunting): picked mon.noname-b con 0x7fa57813f990 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa578145c00 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa578140310 con 0x7fa578145c00 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5780b57f0 con 0x7fa5780b5410 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780a9ea0 con 0x7fa57813f990 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:52.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.777+0000 7fa580290640 10 monclient(hunting): authenticate will time out at 1530.939663s 2024-05-15T12:32:52.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.779+0000 7fd42dffb640 1 -- 172.21.15.49:0/3831644811 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fd43404b070 con 0x7fd4380726a0 2024-05-15T12:32:52.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.779+0000 7fd42dffb640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd41c06ce20 0x7fd41c06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.779+0000 7fd43cd9c640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd41c06ce20 0x7fd41c06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.780+0000 7fd43cd9c640 10 monclient: get_auth_request con 0x7fd41c06ce20 auth_method 0 2024-05-15T12:32:52.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.780+0000 7fd42dffb640 10 client.4341.objecter ms_handle_connect 0x7fd41c06ce20 2024-05-15T12:32:52.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.780+0000 7fd43cd9c640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd41c06ce20 0x7fd41c06f2e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd4380b0bb0 tx=0x7fd428006c80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.783+0000 7fd42dffb640 1 -- 172.21.15.49:0/3831644811 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd434030810 con 0x7fd4380726a0 2024-05-15T12:32:52.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.783+0000 7fd42dffb640 10 client.4341.objecter ms_dispatch 0x7fd438000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:52.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.783+0000 7fd42dffb640 3 client.4341.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:52.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.783+0000 7fd42dffb640 3 client.4341.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:52.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.783+0000 7fd42dffb640 20 client.4341.objecter dump_active .. 0 homeless 2024-05-15T12:32:52.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.802+0000 7f867dc47640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:32:52.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.802+0000 7f867dc47640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:32:52.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.803+0000 7f867dc47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:52.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.803+0000 7f867dc47640 1 -- 172.21.15.49:0/548861511 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f86780b6f20 con 0x7f86781353a0 2024-05-15T12:32:52.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.805+0000 7f8d16ffd640 1 -- 172.21.15.49:0/1708148274 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f8d18003b80 con 0x7f8d280b5410 2024-05-15T12:32:52.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.806+0000 7f8d16ffd640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d0806cdd0 0x7f8d0806f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.806+0000 7f8d25d9b640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d0806cdd0 0x7f8d0806f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.806+0000 7f8d25d9b640 10 monclient: get_auth_request con 0x7f8d0806cdd0 auth_method 0 2024-05-15T12:32:52.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.806+0000 7f8d16ffd640 10 client.4301.objecter ms_handle_connect 0x7f8d0806cdd0 2024-05-15T12:32:52.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.806+0000 7f8d25d9b640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d0806cdd0 0x7f8d0806f290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0094a0 tx=0x7f8d1c0093f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.932+0000 7f74aaffd640 1 -- 172.21.15.49:0/1605078804 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f74a4007020 con 0x7f74b40b53e0 2024-05-15T12:32:52.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.933+0000 7f74aaffd640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f749406d1d0 0x7f749406f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:52.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.933+0000 7f74abfff640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f749406d1d0 0x7f749406f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:52.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.933+0000 7f74abfff640 10 monclient: get_auth_request con 0x7f749406d1d0 auth_method 0 2024-05-15T12:32:52.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.934+0000 7f74aaffd640 10 client.4360.objecter ms_handle_connect 0x7f749406d1d0 2024-05-15T12:32:52.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.934+0000 7f74abfff640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f749406d1d0 0x7f749406f690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f749c000f60 tx=0x7f749c007a80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:52.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.936+0000 7f74aaffd640 1 -- 172.21.15.49:0/1605078804 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f74a40050d0 con 0x7f74b40b53e0 2024-05-15T12:32:52.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.936+0000 7f74aaffd640 10 client.4360.objecter ms_dispatch 0x7f74b4000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:52.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.936+0000 7f74aaffd640 3 client.4360.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:52.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.936+0000 7f74aaffd640 3 client.4360.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:52.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.936+0000 7f74aaffd640 20 client.4360.objecter dump_active .. 0 homeless 2024-05-15T12:32:52.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.946+0000 7f74aaffd640 1 -- 172.21.15.49:0/1605078804 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f74b40b7700 con 0x7f74b40b53e0 2024-05-15T12:32:52.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.946+0000 7f74aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:52.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:52.946+0000 7f74aaffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:53.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.037+0000 7f8d16ffd640 1 -- 172.21.15.49:0/1708148274 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8d180561c0 con 0x7f8d280b5410 2024-05-15T12:32:53.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.037+0000 7f8d16ffd640 10 client.4301.objecter ms_dispatch 0x7f8d28000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:53.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.037+0000 7f8d16ffd640 3 client.4301.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:53.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.037+0000 7f8d16ffd640 3 client.4301.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:53.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.037+0000 7f8d16ffd640 20 client.4301.objecter dump_active .. 0 homeless 2024-05-15T12:32:53.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.037+0000 7f8d16ffd640 1 -- 172.21.15.49:0/1708148274 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8d18020020 con 0x7f8d280b5410 2024-05-15T12:32:53.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.037+0000 7f8d16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:53.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.037+0000 7f8d16ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:53.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.041+0000 7fa57d82b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa57813f990 0x7fa57813fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:53.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.042+0000 7fa57d82b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa57813f990 0x7fa57813fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:46338/0 (socket says 172.21.15.49:46338) 2024-05-15T12:32:53.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.042+0000 7fa57d82b640 1 -- 172.21.15.49:0/3243173765 learned_addr learned my addr 172.21.15.49:0/3243173765 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:53.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.042+0000 7fa57d82b640 10 monclient(hunting): get_auth_request con 0x7fa57813f990 auth_method 0 2024-05-15T12:32:53.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.042+0000 7fa57d82b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:53.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.042+0000 7fa57d82b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:53.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.042+0000 7fa57d82b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:53.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.043+0000 7fa57d82b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:53.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.043+0000 7fa57d82b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:53.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.043+0000 7fa57d82b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:53.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.045+0000 7fa57e02c640 1 --2- 172.21.15.49:0/3243173765 >> v2:172.21.15.49:3300/0 conn(0x7fa5780b5410 0x7fa5780a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:53.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.045+0000 7fa57d82b640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-05-15T12:32:53.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.045+0000 7fa57d82b640 1 -- 172.21.15.49:0/3243173765 >> v2:172.21.15.49:3300/0 conn(0x7fa5780b5410 msgr2=0x7fa5780a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:53.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.045+0000 7fa57d82b640 1 --2- 172.21.15.49:0/3243173765 >> v2:172.21.15.49:3300/0 conn(0x7fa5780b5410 0x7fa5780a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.045+0000 7fa57d82b640 1 -- 172.21.15.49:0/3243173765 >> v1:172.21.15.49:6789/0 conn(0x7fa578145c00 legacy=0x7fa578146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:53.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.045+0000 7fa57d82b640 10 monclient: _finish_hunting 0 2024-05-15T12:32:53.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.045+0000 7fa57d82b640 1 monclient: found mon.noname-b 2024-05-15T12:32:53.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.045+0000 7fa57d82b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:53.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.045+0000 7fa57d82b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:53.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.045+0000 7fa57d82b640 1 -- 172.21.15.49:0/3243173765 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5780ab470 con 0x7fa57813f990 2024-05-15T12:32:53.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.046+0000 7f865ffff640 1 -- 172.21.15.49:0/548861511 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f866c07afc0 con 0x7f86781353a0 2024-05-15T12:32:53.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.046+0000 7f865ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-15T12:32:53.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.046+0000 7f865ffff640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:53.036 INFO:teuthology.orchestra.run.smithi049.stdout:73014444040 2024-05-15T12:32:53.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.048+0000 7fa57d82b640 1 --2- 172.21.15.49:0/3243173765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa57813f990 0x7fa57813fdd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa568009870 tx=0x7fa56802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa7b677a8da36bfa server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:53.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.049+0000 7f867dc47640 10 librados: watch_flush enter 2024-05-15T12:32:53.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.049+0000 7f867dc47640 10 librados: watch_flush exit 2024-05-15T12:32:53.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.049+0000 7f867dc47640 20 client.4326.objecter shutdown clearing up homeless session... 2024-05-15T12:32:53.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.049+0000 7f867dc47640 10 client.4326.objecter successfully canceled tick 2024-05-15T12:32:53.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.049+0000 7f867dc47640 1 -- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f866406ce50 msgr2=0x7f866406f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.049+0000 7f867dc47640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f866406ce50 0x7f866406f310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8668009710 tx=0x7f8668006d00 comp rx=0 tx=0).stop 2024-05-15T12:32:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.049+0000 7f867dc47640 10 monclient: shutdown 2024-05-15T12:32:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.049+0000 7f867dc47640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.049+0000 7f867dc47640 1 -- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86781353a0 msgr2=0x7f8678135820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.049+0000 7f867dc47640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86781353a0 0x7f8678135820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f866c02bbe0 tx=0x7f866c000fb0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.050+0000 7f867dc47640 1 -- 172.21.15.49:0/548861511 shutdown_connections 2024-05-15T12:32:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.050+0000 7f867dc47640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86780b6160 0x7f86780b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.050+0000 7f867dc47640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86781353a0 0x7f8678135820 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f866c02bbe0 tx=0x7f866c000fb0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.050+0000 7f867dc47640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f866406ce50 0x7f866406f310 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.050+0000 7f867dc47640 1 --2- 172.21.15.49:0/548861511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8660004800 0x7f86780b88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.050+0000 7f867dc47640 1 -- 172.21.15.49:0/548861511 >> 172.21.15.49:0/548861511 conn(0x7f8678066f00 msgr2=0x7f8678138d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:53.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.052+0000 7f867dc47640 1 -- 172.21.15.49:0/548861511 shutdown_connections 2024-05-15T12:32:53.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.052+0000 7f867dc47640 1 -- 172.21.15.49:0/548861511 wait complete. 2024-05-15T12:32:53.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7f867dc47640 1 librados: shutdown 2024-05-15T12:32:53.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.053+0000 7fa57d02a640 1 -- 172.21.15.49:0/3243173765 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5680317c0 con 0x7fa57813f990 2024-05-15T12:32:53.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa57d02a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:53.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa57d02a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:53.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa57d02a640 10 monclient: dump: 2024-05-15T12:32:53.044 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:53.044 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:53.044 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa57d02a640 10 monclient: _finish_auth 0 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa57d02a640 10 monclient: _check_auth_tickets 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa57d02a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa580290640 5 monclient: authenticate success, global_id 4304 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa580290640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa57d02a640 1 -- 172.21.15.49:0/3243173765 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa568037070 con 0x7fa57813f990 2024-05-15T12:32:53.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa57d02a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:53.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa580290640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:53.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.054+0000 7fa580290640 4 set_mon_vals no callback set 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.055+0000 7fa580290640 10 monclient: shutdown 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.055+0000 7fa580290640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.055+0000 7fa580290640 1 -- 172.21.15.49:0/3243173765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa57813f990 msgr2=0x7fa57813fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.055+0000 7fa580290640 1 --2- 172.21.15.49:0/3243173765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa57813f990 0x7fa57813fdd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa568009870 tx=0x7fa56802b860 comp rx=0 tx=0).stop 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.055+0000 7fa580290640 1 -- 172.21.15.49:0/3243173765 shutdown_connections 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.055+0000 7fa580290640 1 --2- 172.21.15.49:0/3243173765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa57813f990 0x7fa57813fdd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.055+0000 7fa580290640 1 --2- 172.21.15.49:0/3243173765 >> v2:172.21.15.49:3300/0 conn(0x7fa5780b5410 0x7fa5780a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.055+0000 7fa580290640 1 -- 172.21.15.49:0/3243173765 >> 172.21.15.49:0/3243173765 conn(0x7fa578066f00 msgr2=0x7fa578067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.056+0000 7fa580290640 1 -- 172.21.15.49:0/3243173765 shutdown_connections 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.056+0000 7fa580290640 1 -- 172.21.15.49:0/3243173765 wait complete. 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.057+0000 7fa580290640 10 monclient: build_initial_monmap 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.057+0000 7fa580290640 10 monclient: monmap: 2024-05-15T12:32:53.047 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:53.048 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:53.048 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:53.048 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:53.048 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:53.048 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:53.048 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:53.048 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:53.048 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:53.048 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:53.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.057+0000 7fa580290640 1 librados: starting msgr at 2024-05-15T12:32:53.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.057+0000 7fa580290640 1 librados: starting objecter 2024-05-15T12:32:53.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.057+0000 7fa580290640 1 Processor -- start 2024-05-15T12:32:53.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.057+0000 7fa580290640 1 -- start start 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.057+0000 7fa580290640 1 librados: setting wanted keys 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.057+0000 7fa580290640 1 librados: calling monclient init 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.057+0000 7fa580290640 10 monclient: init 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5780b5410 0x7fa578142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 10 monclient(hunting): picked mon.noname-a con 0x7fa5780b5410 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa57813f990 0x7fa578140500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 10 monclient(hunting): picked mon.noname-b con 0x7fa57813f990 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa578145c00 0x7fa5781db130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 10 monclient(hunting): picked mon.noname-c con 0x7fa578145c00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780b4aa0 con 0x7fa5780b5410 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:53.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780b4920 con 0x7fa57813f990 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57814cf30 con 0x7fa578145c00 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.058+0000 7fa580290640 10 monclient(hunting): authenticate will time out at 1531.221077s 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57d82b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa57813f990 0x7fa578140500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57e82d640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa578145c00 0x7fa5781db130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57d82b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa57813f990 0x7fa578140500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35374/0 (socket says 172.21.15.49:35374) 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57d82b640 1 -- 172.21.15.49:0/828168002 learned_addr learned my addr 172.21.15.49:0/828168002 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57d82b640 10 monclient(hunting): get_auth_request con 0x7fa57813f990 auth_method 0 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57d82b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57d82b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57d82b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57e82d640 10 monclient(hunting): get_auth_request con 0x7fa578145c00 auth_method 0 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57e82d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:53.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57e82d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:53.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.059+0000 7fa57e82d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:53.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.060+0000 7fa57d82b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:53.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.060+0000 7fa57d82b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:53.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.060+0000 7fa57d82b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:53.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.060+0000 7fa57e82d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:53.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.060+0000 7fa57e82d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:53.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.060+0000 7fa57e82d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:53.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fd42dffb640 1 -- 172.21.15.49:0/3831644811 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd434052020 con 0x7fd4380726a0 2024-05-15T12:32:53.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fd42dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:53.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fd42dffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:53.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fa57e82d640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-05-15T12:32:53.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fa57e82d640 1 -- 172.21.15.49:0/828168002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa57813f990 msgr2=0x7fa578140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fa57e82d640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa57813f990 0x7fa578140500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fa57e82d640 1 -- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5780b5410 msgr2=0x7fa578142da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fa57e82d640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5780b5410 0x7fa578142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fa57e82d640 10 monclient: _finish_hunting 0 2024-05-15T12:32:53.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fa57e82d640 1 monclient: found mon.noname-c 2024-05-15T12:32:53.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fa57e82d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:53.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fa57e82d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T12:32:53.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.063+0000 7fa57e82d640 1 -- 172.21.15.49:0/828168002 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578141de0 con 0x7fa578145c00 2024-05-15T12:32:53.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.067+0000 7fa57e82d640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa578145c00 0x7fa5781db130 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa57000be20 tx=0x7fa57000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:53.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.067+0000 7fa56effd640 10 client.?.objecter ms_handle_connect 0x7fa578145c00 2024-05-15T12:32:53.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.067+0000 7fa56effd640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:53.064 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444040 for osd.2 2024-05-15T12:32:53.064 DEBUG:teuthology.parallel:result is None 2024-05-15T12:32:53.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 1 -- 172.21.15.49:0/828168002 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570019070 con 0x7fa578145c00 2024-05-15T12:32:53.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:53.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:53.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 10 monclient: dump: 2024-05-15T12:32:53.066 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:53.066 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:53.066 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 10 monclient: _finish_auth 0 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 1 -- 172.21.15.49:0/828168002 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57000cdf0 con 0x7fa578145c00 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa580290640 5 monclient: authenticate success, global_id 4307 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa580290640 10 monclient: _renew_subs 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa580290640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:53.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa580290640 1 -- 172.21.15.49:0/828168002 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa578142050 con 0x7fa578145c00 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 1 -- 172.21.15.49:0/828168002 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570014680 con 0x7fa578145c00 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa56effd640 10 monclient: dump: 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:53.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa57c829640 4 set_mon_vals no callback set 2024-05-15T12:32:53.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa580290640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:53.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa580290640 10 monclient: _renew_subs 2024-05-15T12:32:53.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa580290640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:53.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa580290640 1 -- 172.21.15.49:0/828168002 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa578139000 con 0x7fa578145c00 2024-05-15T12:32:53.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.072+0000 7fa580290640 1 librados: init done 2024-05-15T12:32:53.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.073+0000 7fa580290640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:53.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.073+0000 7fa580290640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:53.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.073+0000 7fa580290640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:53.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.073+0000 7fa580290640 1 -- 172.21.15.49:0/828168002 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57804f0e0 con 0x7fa578145c00 2024-05-15T12:32:53.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.144+0000 7f74a8ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:32:53.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.144+0000 7f74a8ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:32:53.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.144+0000 7f74a8ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:53.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.144+0000 7f74a8ff9640 1 -- 172.21.15.49:0/1605078804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f74b404f0e0 con 0x7f74b40b53e0 2024-05-15T12:32:53.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.148+0000 7f74aaffd640 1 -- 172.21.15.49:0/1605078804 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f74a4057db0 con 0x7f74b40b53e0 2024-05-15T12:32:53.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.148+0000 7f74aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-15T12:32:53.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.148+0000 7f74aaffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:53.138 INFO:teuthology.orchestra.run.smithi049.stdout:81604378633 2024-05-15T12:32:53.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.149+0000 7f74a8ff9640 10 librados: watch_flush enter 2024-05-15T12:32:53.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.149+0000 7f74a8ff9640 10 librados: watch_flush exit 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.149+0000 7f74a8ff9640 20 client.4360.objecter shutdown clearing up homeless session... 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.149+0000 7f74a8ff9640 10 client.4360.objecter successfully canceled tick 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 -- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f749406d1d0 msgr2=0x7f749406f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f749406d1d0 0x7f749406f690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f749c000f60 tx=0x7f749c007a80 comp rx=0 tx=0).stop 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 10 monclient: shutdown 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 -- 172.21.15.49:0/1605078804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b40b53e0 msgr2=0x7f74b41db710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b40b53e0 0x7f74b41db710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f74a400c430 tx=0x7f74a400c8f0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 -- 172.21.15.49:0/1605078804 shutdown_connections 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74b414b390 0x7f74b4132840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74b4142870 0x7f74b4132300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f749406d1d0 0x7f749406f690 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 --2- 172.21.15.49:0/1605078804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74b40b53e0 0x7f74b41db710 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 -- 172.21.15.49:0/1605078804 >> 172.21.15.49:0/1605078804 conn(0x7f74b4066ed0 msgr2=0x7f74b40672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:53.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 -- 172.21.15.49:0/1605078804 shutdown_connections 2024-05-15T12:32:53.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 -- 172.21.15.49:0/1605078804 wait complete. 2024-05-15T12:32:53.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.150+0000 7f74a8ff9640 1 librados: shutdown 2024-05-15T12:32:53.152 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378633 got 81604378633 for osd.5 2024-05-15T12:32:53.152 DEBUG:teuthology.parallel:result is None 2024-05-15T12:32:53.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.234+0000 7f8d2cdf8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:32:53.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.234+0000 7f8d2cdf8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:32:53.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.234+0000 7f8d2cdf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:53.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.234+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1708148274 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8d28146ad0 con 0x7f8d280b5410 2024-05-15T12:32:53.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.258+0000 7fd43e59f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:32:53.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.258+0000 7fd43e59f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:32:53.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.258+0000 7fd43e59f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T12:32:53.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.258+0000 7fd43e59f640 1 -- 172.21.15.49:0/3831644811 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd4381db1f0 con 0x7fd4380726a0 2024-05-15T12:32:53.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.262+0000 7fd42dffb640 1 -- 172.21.15.49:0/3831644811 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd43407b920 con 0x7fd4380726a0 2024-05-15T12:32:53.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.262+0000 7fd42dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-15T12:32:53.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.262+0000 7fd42dffb640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:53.253 INFO:teuthology.orchestra.run.smithi049.stdout:81604378634 2024-05-15T12:32:53.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.264+0000 7fd43e59f640 10 librados: watch_flush enter 2024-05-15T12:32:53.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.264+0000 7fd43e59f640 10 librados: watch_flush exit 2024-05-15T12:32:53.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.264+0000 7fd43e59f640 20 client.4341.objecter shutdown clearing up homeless session... 2024-05-15T12:32:53.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.264+0000 7fd43e59f640 10 client.4341.objecter successfully canceled tick 2024-05-15T12:32:53.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 -- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd41c06ce20 msgr2=0x7fd41c06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd41c06ce20 0x7fd41c06f2e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd4380b0bb0 tx=0x7fd428006c80 comp rx=0 tx=0).stop 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 10 monclient: shutdown 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 -- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 msgr2=0x7fd4380ad8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 0x7fd4380ad8a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd43402b9e0 tx=0x7fd434004730 comp rx=0 tx=0).stop 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 -- 172.21.15.49:0/3831644811 shutdown_connections 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4380ae5e0 0x7fd4381db9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd41c06ce20 0x7fd41c06f2e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd438072a80 0x7fd4381db490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 --2- 172.21.15.49:0/3831644811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4380726a0 0x7fd4380ad8a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 -- 172.21.15.49:0/3831644811 >> 172.21.15.49:0/3831644811 conn(0x7fd43806af80 msgr2=0x7fd4380b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 -- 172.21.15.49:0/3831644811 shutdown_connections 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 -- 172.21.15.49:0/3831644811 wait complete. 2024-05-15T12:32:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.265+0000 7fd43e59f640 1 librados: shutdown 2024-05-15T12:32:53.266 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378634 got 81604378634 for osd.7 2024-05-15T12:32:53.266 DEBUG:teuthology.parallel:result is None 2024-05-15T12:32:53.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.353+0000 7fa56effd640 1 -- 172.21.15.49:0/828168002 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7fa57001c070 con 0x7fa578145c00 2024-05-15T12:32:53.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.353+0000 7fa56effd640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa55c06ce20 0x7fa55c06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:53.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.353+0000 7fa57e02c640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa55c06ce20 0x7fa55c06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:53.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.353+0000 7fa57e02c640 10 monclient: get_auth_request con 0x7fa55c06ce20 auth_method 0 2024-05-15T12:32:53.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.354+0000 7fa56effd640 10 client.4307.objecter ms_handle_connect 0x7fa55c06ce20 2024-05-15T12:32:53.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.354+0000 7fa57e02c640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa55c06ce20 0x7fa55c06f2e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa574006fd0 tx=0x7fa574008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:53.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.369+0000 7fa56effd640 1 -- 172.21.15.49:0/828168002 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa570056d60 con 0x7fa578145c00 2024-05-15T12:32:53.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.369+0000 7fa56effd640 10 client.4307.objecter ms_dispatch 0x7fa578000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:53.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.369+0000 7fa56effd640 3 client.4307.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:53.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.369+0000 7fa56effd640 3 client.4307.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:53.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.369+0000 7fa56effd640 20 client.4307.objecter dump_active .. 0 homeless 2024-05-15T12:32:53.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.369+0000 7fa56effd640 1 -- 172.21.15.49:0/828168002 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa570023020 con 0x7fa578145c00 2024-05-15T12:32:53.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.369+0000 7fa56effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:53.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.369+0000 7fa56effd640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.378+0000 7f8d16ffd640 1 -- 172.21.15.49:0/1708148274 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8d18058820 con 0x7f8d280b5410 2024-05-15T12:32:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.378+0000 7f8d16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-15T12:32:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.378+0000 7f8d16ffd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:53.368 INFO:teuthology.orchestra.run.smithi049.stdout:90194313225 2024-05-15T12:32:53.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.380+0000 7f8d2cdf8640 10 librados: watch_flush enter 2024-05-15T12:32:53.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.380+0000 7f8d2cdf8640 10 librados: watch_flush exit 2024-05-15T12:32:53.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.380+0000 7f8d2cdf8640 20 client.4301.objecter shutdown clearing up homeless session... 2024-05-15T12:32:53.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.380+0000 7f8d2cdf8640 10 client.4301.objecter successfully canceled tick 2024-05-15T12:32:53.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.380+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d0806cdd0 msgr2=0x7f8d0806f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.380+0000 7f8d2cdf8640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d0806cdd0 0x7f8d0806f290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0094a0 tx=0x7f8d1c0093f0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.380+0000 7f8d2cdf8640 10 monclient: shutdown 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.380+0000 7f8d2cdf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.380+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d280b5410 msgr2=0x7f8d28134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.380+0000 7f8d2cdf8640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d280b5410 0x7f8d28134c80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800c410 tx=0x7f8d1800c8d0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.381+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1708148274 shutdown_connections 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.381+0000 7f8d2cdf8640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8d28145c00 0x7f8d28141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.381+0000 7f8d2cdf8640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f8d0806cdd0 0x7f8d0806f290 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.381+0000 7f8d2cdf8640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d281400a0 0x7f8d28142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.381+0000 7f8d2cdf8640 1 --2- 172.21.15.49:0/1708148274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8d280b5410 0x7f8d28134c80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.381+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1708148274 >> 172.21.15.49:0/1708148274 conn(0x7f8d28066f00 msgr2=0x7f8d2813ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.381+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1708148274 shutdown_connections 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.381+0000 7f8d2cdf8640 1 -- 172.21.15.49:0/1708148274 wait complete. 2024-05-15T12:32:53.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.381+0000 7f8d2cdf8640 1 librados: shutdown 2024-05-15T12:32:53.382 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313225 for osd.1 2024-05-15T12:32:53.383 DEBUG:teuthology.parallel:result is None 2024-05-15T12:32:53.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.557+0000 7fa580290640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:32:53.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.557+0000 7fa580290640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:32:53.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.557+0000 7fa580290640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T12:32:53.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.557+0000 7fa580290640 1 -- 172.21.15.49:0/828168002 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa578146ad0 con 0x7fa578145c00 2024-05-15T12:32:53.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.563+0000 7fa56effd640 1 -- 172.21.15.49:0/828168002 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa570058cc0 con 0x7fa578145c00 2024-05-15T12:32:53.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.563+0000 7fa56effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-15T12:32:53.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.563+0000 7fa56effd640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:53.553 INFO:teuthology.orchestra.run.smithi049.stdout:73014444040 2024-05-15T12:32:53.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 10 librados: watch_flush enter 2024-05-15T12:32:53.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 10 librados: watch_flush exit 2024-05-15T12:32:53.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 20 client.4307.objecter shutdown clearing up homeless session... 2024-05-15T12:32:53.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 10 client.4307.objecter successfully canceled tick 2024-05-15T12:32:53.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 1 -- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa55c06ce20 msgr2=0x7fa55c06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa55c06ce20 0x7fa55c06f2e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa574006fd0 tx=0x7fa574008040 comp rx=0 tx=0).stop 2024-05-15T12:32:53.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 10 monclient: shutdown 2024-05-15T12:32:53.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:53.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 1 -- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa578145c00 msgr2=0x7fa5781db130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa578145c00 0x7fa5781db130 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa57000be20 tx=0x7fa57000bf10 comp rx=0 tx=0).stop 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 1 -- 172.21.15.49:0/828168002 shutdown_connections 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa578145c00 0x7fa5781db130 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fa55c06ce20 0x7fa55c06f2e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa57813f990 0x7fa578140500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 1 --2- 172.21.15.49:0/828168002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5780b5410 0x7fa578142da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.565+0000 7fa580290640 1 -- 172.21.15.49:0/828168002 >> 172.21.15.49:0/828168002 conn(0x7fa578066f00 msgr2=0x7fa5780a9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.566+0000 7fa580290640 1 -- 172.21.15.49:0/828168002 shutdown_connections 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.566+0000 7fa580290640 1 -- 172.21.15.49:0/828168002 wait complete. 2024-05-15T12:32:53.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.566+0000 7fa580290640 1 librados: shutdown 2024-05-15T12:32:53.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444040 for osd.0 2024-05-15T12:32:53.568 DEBUG:teuthology.parallel:result is None 2024-05-15T12:32:53.568 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-15T12:32:53.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T12:32:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.701+0000 7f059a685640 10 monclient: get_monmap_and_config 2024-05-15T12:32:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.701+0000 7f059a685640 10 monclient: build_initial_monmap 2024-05-15T12:32:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.701+0000 7f059a685640 10 monclient: monmap: 2024-05-15T12:32:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:53.702844+0000 2024-05-15T12:32:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:53.702844+0000 2024-05-15T12:32:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:53.692 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:53.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.702+0000 7f059a685640 1 Processor -- start 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.702+0000 7f059a685640 1 -- start start 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.702+0000 7f059a685640 10 monclient: init 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05940b53e0 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0594142870 0x7f0594142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 10 monclient(hunting): picked mon.noname-c con 0x7f0594142870 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f05941431f0 0x7f059413f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 10 monclient(hunting): picked mon.noname-a con 0x7f05941431f0 addr v2:172.21.15.49:3300/0 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05940ab230 con 0x7f05940b53e0 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:53.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05940b57c0 con 0x7f05941431f0 2024-05-15T12:32:53.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:53.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05940aa230 con 0x7f0594142870 2024-05-15T12:32:53.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:53.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.703+0000 7f059a685640 10 monclient(hunting): authenticate will time out at 1531.865790s 2024-05-15T12:32:53.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.704+0000 7f05937fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0594142870 0x7f0594142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.704+0000 7f05937fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0594142870 0x7f0594142cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35376/0 (socket says 172.21.15.49:35376) 2024-05-15T12:32:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.704+0000 7f05937fe640 1 -- 172.21.15.49:0/2509835298 learned_addr learned my addr 172.21.15.49:0/2509835298 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.704+0000 7f05937fe640 10 monclient(hunting): get_auth_request con 0x7f0594142870 auth_method 0 2024-05-15T12:32:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.704+0000 7f05937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.704+0000 7f05937fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.704+0000 7f05937fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.705+0000 7f05937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.705+0000 7f05937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.705+0000 7f05937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.707+0000 7f05937fe640 10 monclient(hunting): handle_auth_done global_id 4366 payload 274 2024-05-15T12:32:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.707+0000 7f05937fe640 1 -- 172.21.15.49:0/2509835298 >> v2:172.21.15.49:3300/0 conn(0x7f05941431f0 msgr2=0x7f059413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.707+0000 7f05937fe640 1 --2- 172.21.15.49:0/2509835298 >> v2:172.21.15.49:3300/0 conn(0x7f05941431f0 0x7f059413f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.707+0000 7f05937fe640 1 -- 172.21.15.49:0/2509835298 >> v1:172.21.15.88:6789/0 conn(0x7f05940b53e0 legacy=0x7f05940a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.707+0000 7f05937fe640 10 monclient: _finish_hunting 0 2024-05-15T12:32:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.707+0000 7f05937fe640 1 monclient: found mon.noname-c 2024-05-15T12:32:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.707+0000 7f05937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:53.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.707+0000 7f05937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:53.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.707+0000 7f05937fe640 1 -- 172.21.15.49:0/2509835298 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940ab510 con 0x7f0594142870 2024-05-15T12:32:53.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.710+0000 7f05937fe640 1 --2- 172.21.15.49:0/2509835298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0594142870 0x7f0594142cb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f05940b4660 tx=0x7f058802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a65749e94010502a server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:53.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 1 -- 172.21.15.49:0/2509835298 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588035020 con 0x7f0594142870 2024-05-15T12:32:53.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:53.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:53.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 10 monclient: dump: 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 10 monclient: _finish_auth 0 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 10 monclient: _check_auth_tickets 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:53.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 1 -- 172.21.15.49:0/2509835298 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0588031870 con 0x7f0594142870 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 1 -- 172.21.15.49:0/2509835298 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588031c40 con 0x7f0594142870 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f059a685640 5 monclient: authenticate success, global_id 4366 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f059a685640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f059a685640 4 set_mon_vals no callback set 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.714+0000 7f0592ffd640 10 monclient: dump: 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:53.707 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:53.707 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:53.707 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:53.707 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:53.707 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:53.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.715+0000 7f059a685640 10 monclient: shutdown 2024-05-15T12:32:53.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.715+0000 7f059a685640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:53.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.715+0000 7f059a685640 1 -- 172.21.15.49:0/2509835298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0594142870 msgr2=0x7f0594142cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.715+0000 7f059a685640 1 --2- 172.21.15.49:0/2509835298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0594142870 0x7f0594142cb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f05940b4660 tx=0x7f058802b910 comp rx=0 tx=0).stop 2024-05-15T12:32:53.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.715+0000 7f059a685640 1 -- 172.21.15.49:0/2509835298 shutdown_connections 2024-05-15T12:32:53.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.715+0000 7f059a685640 1 --2- 172.21.15.49:0/2509835298 >> v2:172.21.15.49:3300/0 conn(0x7f05941431f0 0x7f059413f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.715+0000 7f059a685640 1 --2- 172.21.15.49:0/2509835298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0594142870 0x7f0594142cb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.715+0000 7f059a685640 1 -- 172.21.15.49:0/2509835298 >> 172.21.15.49:0/2509835298 conn(0x7f0594066ed0 msgr2=0x7f05940672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.715+0000 7f059a685640 1 -- 172.21.15.49:0/2509835298 shutdown_connections 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.715+0000 7f059a685640 1 -- 172.21.15.49:0/2509835298 wait complete. 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.716+0000 7f059a685640 10 monclient: build_initial_monmap 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.716+0000 7f059a685640 10 monclient: monmap: 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.717+0000 7f059a685640 1 librados: starting msgr at 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.717+0000 7f059a685640 1 librados: starting objecter 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.717+0000 7f059a685640 1 Processor -- start 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.717+0000 7f059a685640 1 -- start start 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.717+0000 7f059a685640 1 librados: setting wanted keys 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.717+0000 7f059a685640 1 librados: calling monclient init 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.717+0000 7f059a685640 10 monclient: init 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05940b53e0 0x7f05941db190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 10 monclient(hunting): picked mon.noname-c con 0x7f05940b53e0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:53.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0594142870 0x7f059413a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 10 monclient(hunting): picked mon.noname-a con 0x7f0594142870 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05941431f0 0x7f059413aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 10 monclient(hunting): picked mon.noname-b con 0x7f05941431f0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059413ff30 con 0x7f0594142870 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059413fdb0 con 0x7f05941431f0 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05941400b0 con 0x7f05940b53e0 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.718+0000 7f059a685640 10 monclient(hunting): authenticate will time out at 1531.880875s 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0598c22640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05941431f0 0x7f059413aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0593fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05940b53e0 0x7f05941db190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0598c22640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05941431f0 0x7f059413aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35378/0 (socket says 172.21.15.49:35378) 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0598c22640 1 -- 172.21.15.49:0/3171203652 learned_addr learned my addr 172.21.15.49:0/3171203652 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0598c22640 10 monclient(hunting): get_auth_request con 0x7f05941431f0 auth_method 0 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0598c22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0598c22640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0598c22640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0593fff640 10 monclient(hunting): get_auth_request con 0x7f05940b53e0 auth_method 0 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0593fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0593fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0593fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0598c22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.719+0000 7f0598c22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.720+0000 7f0598c22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.720+0000 7f0593fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.720+0000 7f0593fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.720+0000 7f0593fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:53.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.722+0000 7f0598c22640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-05-15T12:32:53.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.722+0000 7f0598c22640 1 -- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05940b53e0 msgr2=0x7f05941db190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.722+0000 7f0598c22640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05940b53e0 0x7f05941db190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.722+0000 7f0598c22640 1 -- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0594142870 msgr2=0x7f059413a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:53.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.722+0000 7f0598c22640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0594142870 0x7f059413a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:53.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.722+0000 7f0598c22640 10 monclient: _finish_hunting 0 2024-05-15T12:32:53.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.722+0000 7f0598c22640 1 monclient: found mon.noname-b 2024-05-15T12:32:53.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.722+0000 7f0598c22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:53.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.722+0000 7f0598c22640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:53.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.722+0000 7f0598c22640 1 -- 172.21.15.49:0/3171203652 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940adc50 con 0x7f05941431f0 2024-05-15T12:32:53.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.724+0000 7f0598c22640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05941431f0 0x7f059413aa60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f058c00b9a0 tx=0x7f058c00be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:53.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.724+0000 7f0590ff9640 10 client.?.objecter ms_handle_connect 0x7f05941431f0 2024-05-15T12:32:53.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.724+0000 7f0590ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:53.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 1 -- 172.21.15.49:0/3171203652 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c019070 con 0x7f05941431f0 2024-05-15T12:32:53.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:53.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:53.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 10 monclient: dump: 2024-05-15T12:32:53.719 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:53.719 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:53.719 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.719 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.719 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 10 monclient: _finish_auth 0 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 10 monclient: _check_auth_tickets 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 1 -- 172.21.15.49:0/3171203652 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c00cbc0 con 0x7f05941431f0 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 1 -- 172.21.15.49:0/3171203652 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c014440 con 0x7f05941431f0 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:53.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f0590ff9640 10 monclient: dump: 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f059a685640 5 monclient: authenticate success, global_id 4369 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f059a685640 10 monclient: _renew_subs 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f059a685640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:53.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f059a685640 1 -- 172.21.15.49:0/3171203652 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0594120ab0 con 0x7f05941431f0 2024-05-15T12:32:53.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f05927fc640 4 set_mon_vals no callback set 2024-05-15T12:32:53.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f059a685640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:53.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f059a685640 10 monclient: _renew_subs 2024-05-15T12:32:53.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f059a685640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:53.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f059a685640 1 -- 172.21.15.49:0/3171203652 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05940b0630 con 0x7f05941431f0 2024-05-15T12:32:53.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.728+0000 7f059a685640 1 librados: init done 2024-05-15T12:32:53.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.729+0000 7f059a685640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:53.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.729+0000 7f059a685640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:53.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.729+0000 7f059a685640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:53.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:53.729+0000 7f059a685640 1 -- 172.21.15.49:0/3171203652 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05940b21c0 con 0x7f05941431f0 2024-05-15T12:32:54.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.020+0000 7f0590ff9640 1 -- 172.21.15.49:0/3171203652 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f058c01c070 con 0x7f05941431f0 2024-05-15T12:32:54.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.021+0000 7f0590ff9640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f058006ce20 0x7f058006f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.021+0000 7f0593fff640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f058006ce20 0x7f058006f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.021+0000 7f0593fff640 10 monclient: get_auth_request con 0x7f058006ce20 auth_method 0 2024-05-15T12:32:54.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.021+0000 7f0593fff640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f058006ce20 0x7f058006f2e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0584000f60 tx=0x7f0584009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:54.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.021+0000 7f0590ff9640 10 client.4369.objecter ms_handle_connect 0x7f058006ce20 2024-05-15T12:32:54.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.023+0000 7f0590ff9640 1 -- 172.21.15.49:0/3171203652 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f058c056bd0 con 0x7f05941431f0 2024-05-15T12:32:54.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.023+0000 7f0590ff9640 10 client.4369.objecter ms_dispatch 0x7f0594000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:54.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.023+0000 7f0590ff9640 3 client.4369.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:54.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.023+0000 7f0590ff9640 3 client.4369.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:54.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.023+0000 7f0590ff9640 20 client.4369.objecter dump_active .. 0 homeless 2024-05-15T12:32:54.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.033+0000 7f0590ff9640 1 -- 172.21.15.49:0/3171203652 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f05940b21c0 con 0x7f05941431f0 2024-05-15T12:32:54.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.033+0000 7f0590ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:54.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.033+0000 7f0590ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:54.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.236+0000 7f059a685640 1 -- 172.21.15.49:0/3171203652 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05941208b0 con 0x7f058006ce20 2024-05-15T12:32:54.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.238+0000 7f0590ff9640 1 -- 172.21.15.49:0/3171203652 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+35508 (secure 0 0 0) 0x7f05941208b0 con 0x7f058006ce20 2024-05-15T12:32:54.228 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:32:54.229 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-05-15T12:32:54.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 10 librados: watch_flush enter 2024-05-15T12:32:54.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 10 librados: watch_flush exit 2024-05-15T12:32:54.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 20 client.4369.objecter shutdown clearing up homeless session... 2024-05-15T12:32:54.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 10 client.4369.objecter successfully canceled tick 2024-05-15T12:32:54.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 1 -- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f058006ce20 msgr2=0x7f058006f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f058006ce20 0x7f058006f2e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0584000f60 tx=0x7f0584009040 comp rx=0 tx=0).stop 2024-05-15T12:32:54.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 10 monclient: shutdown 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 1 -- 172.21.15.49:0/3171203652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05941431f0 msgr2=0x7f059413aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05941431f0 0x7f059413aa60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f058c00b9a0 tx=0x7f058c00be60 comp rx=0 tx=0).stop 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 1 -- 172.21.15.49:0/3171203652 shutdown_connections 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f058006ce20 0x7f058006f2e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.240+0000 7f059a685640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05941431f0 0x7f059413aa60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.241+0000 7f059a685640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0594142870 0x7f059413a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.241+0000 7f059a685640 1 --2- 172.21.15.49:0/3171203652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05940b53e0 0x7f05941db190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.241+0000 7f059a685640 1 -- 172.21.15.49:0/3171203652 >> 172.21.15.49:0/3171203652 conn(0x7f0594066ed0 msgr2=0x7f059413f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.241+0000 7f059a685640 1 -- 172.21.15.49:0/3171203652 shutdown_connections 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.241+0000 7f059a685640 1 -- 172.21.15.49:0/3171203652 wait complete. 2024-05-15T12:32:54.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.241+0000 7f059a685640 1 librados: shutdown 2024-05-15T12:32:54.242 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-05-15T12:32:52.286836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218788,"kb_used_data":2600,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755932,"statfs":{"total":773094113280,"available":772870074368,"internally_reserved":0,"allocated":2662400,"data_stored":1376208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.176965"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.314795+0000","last_change":"2024-05-15T12:32:48.258451+0000","last_active":"2024-05-15T12:32:48.314795+0000","last_peered":"2024-05-15T12:32:48.314795+0000","last_clean":"2024-05-15T12:32:48.314795+0000","last_became_active":"2024-05-15T12:32:48.252725+0000","last_became_peered":"2024-05-15T12:32:48.252725+0000","last_unstale":"2024-05-15T12:32:48.314795+0000","last_undegraded":"2024-05-15T12:32:48.314795+0000","last_fullsized":"2024-05-15T12:32:48.314795+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:37:05.959841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:32:49.380976+0000","last_change":"2024-05-15T12:32:48.309674+0000","last_active":"2024-05-15T12:32:49.380976+0000","last_peered":"2024-05-15T12:32:49.380976+0000","last_clean":"2024-05-15T12:32:49.380976+0000","last_became_active":"2024-05-15T12:32:48.283635+0000","last_became_peered":"2024-05-15T12:32:48.283635+0000","last_unstale":"2024-05-15T12:32:49.380976+0000","last_undegraded":"2024-05-15T12:32:49.380976+0000","last_fullsized":"2024-05-15T12:32:49.380976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:07:32.002393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.337225+0000","last_change":"2024-05-15T12:32:47.531833+0000","last_active":"2024-05-15T12:32:48.337225+0000","last_peered":"2024-05-15T12:32:48.337225+0000","last_clean":"2024-05-15T12:32:48.337225+0000","last_became_active":"2024-05-15T12:32:47.526275+0000","last_became_peered":"2024-05-15T12:32:47.526275+0000","last_unstale":"2024-05-15T12:32:48.337225+0000","last_undegraded":"2024-05-15T12:32:48.337225+0000","last_fullsized":"2024-05-15T12:32:48.337225+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:21:07.065217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:32:49.302979+0000","last_change":"2024-05-15T12:32:47.687489+0000","last_active":"2024-05-15T12:32:49.302979+0000","last_peered":"2024-05-15T12:32:49.302979+0000","last_clean":"2024-05-15T12:32:49.302979+0000","last_became_active":"2024-05-15T12:32:47.680498+0000","last_became_peered":"2024-05-15T12:32:47.680498+0000","last_unstale":"2024-05-15T12:32:49.302979+0000","last_undegraded":"2024-05-15T12:32:49.302979+0000","last_fullsized":"2024-05-15T12:32:49.302979+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:09:36.650810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.322083+0000","last_change":"2024-05-15T12:32:47.680657+0000","last_active":"2024-05-15T12:32:48.322083+0000","last_peered":"2024-05-15T12:32:48.322083+0000","last_clean":"2024-05-15T12:32:48.322083+0000","last_became_active":"2024-05-15T12:32:47.670226+0000","last_became_peered":"2024-05-15T12:32:47.670226+0000","last_unstale":"2024-05-15T12:32:48.322083+0000","last_undegraded":"2024-05-15T12:32:48.322083+0000","last_fullsized":"2024-05-15T12:32:48.322083+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:37:30.875058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.515596+0000","last_change":"2024-05-15T12:32:47.463044+0000","last_active":"2024-05-15T12:32:48.515596+0000","last_peered":"2024-05-15T12:32:48.515596+0000","last_clean":"2024-05-15T12:32:48.515596+0000","last_became_active":"2024-05-15T12:32:47.432679+0000","last_became_peered":"2024-05-15T12:32:47.432679+0000","last_unstale":"2024-05-15T12:32:48.515596+0000","last_undegraded":"2024-05-15T12:32:48.515596+0000","last_fullsized":"2024-05-15T12:32:48.515596+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:04:23.139275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.325652+0000","last_change":"2024-05-15T12:32:47.459103+0000","last_active":"2024-05-15T12:32:48.325652+0000","last_peered":"2024-05-15T12:32:48.325652+0000","last_clean":"2024-05-15T12:32:48.325652+0000","last_became_active":"2024-05-15T12:32:47.430194+0000","last_became_peered":"2024-05-15T12:32:47.430194+0000","last_unstale":"2024-05-15T12:32:48.325652+0000","last_undegraded":"2024-05-15T12:32:48.325652+0000","last_fullsized":"2024-05-15T12:32:48.325652+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:30:01.614452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.331314+0000","last_change":"2024-05-15T12:32:47.722217+0000","last_active":"2024-05-15T12:32:48.331314+0000","last_peered":"2024-05-15T12:32:48.331314+0000","last_clean":"2024-05-15T12:32:48.331314+0000","last_became_active":"2024-05-15T12:32:47.716674+0000","last_became_peered":"2024-05-15T12:32:47.716674+0000","last_unstale":"2024-05-15T12:32:48.331314+0000","last_undegraded":"2024-05-15T12:32:48.331314+0000","last_fullsized":"2024-05-15T12:32:48.331314+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:38:58.704500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'50","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:32:50.308588+0000","last_change":"2024-05-15T12:32:43.562093+0000","last_active":"2024-05-15T12:32:50.308588+0000","last_peered":"2024-05-15T12:32:50.308588+0000","last_clean":"2024-05-15T12:32:50.308588+0000","last_became_active":"2024-05-15T12:32:42.889644+0000","last_became_peered":"2024-05-15T12:32:42.889644+0000","last_unstale":"2024-05-15T12:32:50.308588+0000","last_undegraded":"2024-05-15T12:32:50.308588+0000","last_fullsized":"2024-05-15T12:32:50.308588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:30.812143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:30.812143+0000","last_clean_scrub_stamp":"2024-05-15T12:32:30.812143+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:45:22.988060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":672800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":593920,"data_stored":427200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":19,"seq":81604378634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":85052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378633,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":593920,"data_stored":427200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":85052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444040,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":90800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":85052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":85052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":90800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:32:54.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T12:32:54.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.370+0000 7ff00d235640 10 monclient: get_monmap_and_config 2024-05-15T12:32:54.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.370+0000 7ff00d235640 10 monclient: build_initial_monmap 2024-05-15T12:32:54.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.370+0000 7ff00d235640 10 monclient: monmap: 2024-05-15T12:32:54.361 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:54.361 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:54.361 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:54.372153+0000 2024-05-15T12:32:54.361 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:54.372153+0000 2024-05-15T12:32:54.361 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:54.361 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.361 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:54.362 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:54.362 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:54.362 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:54.362 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:54.362 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 1 Processor -- start 2024-05-15T12:32:54.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 1 -- start start 2024-05-15T12:32:54.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 10 monclient: init 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0080b53e0 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff008142870 0x7ff008142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 10 monclient(hunting): picked mon.noname-c con 0x7ff008142870 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0081431f0 addr v1:172.21.15.88:6789/0 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0080ab230 con 0x7ff0080b53e0 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0080ab510 con 0x7ff0081431f0 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0080b57c0 con 0x7ff008142870 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.371+0000 7ff00d235640 10 monclient(hunting): authenticate will time out at 1532.534405s 2024-05-15T12:32:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.372+0000 7ff00659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff008142870 0x7ff008142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.372+0000 7ff006d9d640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7ff0080b53e0 0x7ff0080a99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34688/0 (socket says 172.21.15.49:34688) 2024-05-15T12:32:54.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.372+0000 7ff006d9d640 1 -- 172.21.15.49:0/1609758552 learned_addr learned my addr 172.21.15.49:0/1609758552 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:54.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.373+0000 7ff00659c640 10 monclient(hunting): get_auth_request con 0x7ff008142870 auth_method 0 2024-05-15T12:32:54.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.373+0000 7ff00659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:54.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.373+0000 7ff00659c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:54.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.373+0000 7ff00659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:54.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.374+0000 7ff00659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:54.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.374+0000 7ff00659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:54.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.374+0000 7ff00659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:54.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff005d9b640 1 -- 172.21.15.49:0/1609758552 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265749812 0 0) 0x7ff0080ab230 con 0x7ff0080b53e0 2024-05-15T12:32:54.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff005d9b640 10 monclient(hunting): _negotiate 2024-05-15T12:32:54.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff005d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:54.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff005d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:54.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff005d9b640 10 monclient(hunting): my global_id is 4344 2024-05-15T12:32:54.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff005d9b640 1 -- 172.21.15.49:0/1609758552 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff0003520 con 0x7ff0080b53e0 2024-05-15T12:32:54.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff00659c640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2024-05-15T12:32:54.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff00659c640 1 -- 172.21.15.49:0/1609758552 >> v1:172.21.15.88:6789/0 conn(0x7ff0081431f0 legacy=0x7ff00813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff00659c640 1 -- 172.21.15.49:0/1609758552 >> v1:172.21.15.49:6789/0 conn(0x7ff0080b53e0 legacy=0x7ff0080a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff00659c640 10 monclient: _finish_hunting 0 2024-05-15T12:32:54.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff00659c640 1 monclient: found mon.noname-c 2024-05-15T12:32:54.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff00659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:54.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff00659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T12:32:54.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.375+0000 7ff00659c640 1 -- 172.21.15.49:0/1609758552 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00813fd70 con 0x7ff008142870 2024-05-15T12:32:54.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.379+0000 7ff00659c640 1 --2- 172.21.15.49:0/1609758552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 0x7ff008142cb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feffc0097a0 tx=0x7feffc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=605a763aa4a7c77d server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:54.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 1 -- 172.21.15.49:0/1609758552 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc035020 con 0x7ff008142870 2024-05-15T12:32:54.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:54.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T12:32:54.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 10 monclient: dump: 2024-05-15T12:32:54.374 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:54.374 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 10 monclient: _finish_auth 0 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 10 monclient: _check_auth_tickets 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff00d235640 5 monclient: authenticate success, global_id 4372 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 1 -- 172.21.15.49:0/1609758552 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feffc031830 con 0x7ff008142870 2024-05-15T12:32:54.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 1 -- 172.21.15.49:0/1609758552 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc031c00 con 0x7ff008142870 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff00d235640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff00d235640 4 set_mon_vals no callback set 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.384+0000 7ff005d9b640 10 monclient: dump: 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:54.376 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:54.377 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:54.377 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.385+0000 7ff00d235640 10 monclient: shutdown 2024-05-15T12:32:54.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.385+0000 7ff00d235640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.385+0000 7ff00d235640 1 -- 172.21.15.49:0/1609758552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 msgr2=0x7ff008142cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.385+0000 7ff00d235640 1 --2- 172.21.15.49:0/1609758552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 0x7ff008142cb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feffc0097a0 tx=0x7feffc02b8d0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.385+0000 7ff00d235640 1 -- 172.21.15.49:0/1609758552 shutdown_connections 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.385+0000 7ff00d235640 1 --2- 172.21.15.49:0/1609758552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 0x7ff008142cb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.385+0000 7ff00d235640 1 -- 172.21.15.49:0/1609758552 >> 172.21.15.49:0/1609758552 conn(0x7ff008066ed0 msgr2=0x7ff0080672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.385+0000 7ff00d235640 1 -- 172.21.15.49:0/1609758552 shutdown_connections 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.385+0000 7ff00d235640 1 -- 172.21.15.49:0/1609758552 wait complete. 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.386+0000 7ff00d235640 10 monclient: build_initial_monmap 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.386+0000 7ff00d235640 10 monclient: monmap: 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.386+0000 7ff00d235640 1 librados: starting msgr at 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.386+0000 7ff00d235640 1 librados: starting objecter 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.386+0000 7ff00d235640 1 Processor -- start 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.386+0000 7ff00d235640 1 -- start start 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.386+0000 7ff00d235640 1 librados: setting wanted keys 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.386+0000 7ff00d235640 1 librados: calling monclient init 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.386+0000 7ff00d235640 10 monclient: init 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:32:54.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0080b53e0 0x7ff0080abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 10 monclient(hunting): picked mon.noname-a con 0x7ff0080b53e0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 0x7ff0080ac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 10 monclient(hunting): picked mon.noname-b con 0x7ff008142870 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0081431f0 0x7ff008133e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 10 monclient(hunting): picked mon.noname-c con 0x7ff0081431f0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0080aa230 con 0x7ff0080b53e0 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0081403d0 con 0x7ff008142870 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00814e3b0 con 0x7ff0081431f0 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff00d235640 10 monclient(hunting): authenticate will time out at 1532.549842s 2024-05-15T12:32:54.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.387+0000 7ff006d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0080b53e0 0x7ff0080abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 0x7ff0080ac3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 0x7ff0080ac3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35388/0 (socket says 172.21.15.49:35388) 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00659c640 1 -- 172.21.15.49:0/1635728938 learned_addr learned my addr 172.21.15.49:0/1635728938 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00659c640 10 monclient(hunting): get_auth_request con 0x7ff008142870 auth_method 0 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00659c640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff006d9d640 10 monclient(hunting): get_auth_request con 0x7ff0080b53e0 auth_method 0 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff006d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff006d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff006d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.388+0000 7ff00759e640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0081431f0 0x7ff008133e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.389+0000 7ff006d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:54.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.389+0000 7ff006d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:54.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.389+0000 7ff006d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:54.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.391+0000 7ff00659c640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2024-05-15T12:32:54.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.391+0000 7ff00659c640 1 -- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0081431f0 msgr2=0x7ff008133e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.391+0000 7ff00659c640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0081431f0 0x7ff008133e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.391+0000 7ff00659c640 1 -- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0080b53e0 msgr2=0x7ff0080abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.391+0000 7ff00659c640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0080b53e0 0x7ff0080abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.391+0000 7ff00659c640 10 monclient: _finish_hunting 0 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.391+0000 7ff00659c640 1 monclient: found mon.noname-b 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.391+0000 7ff00659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.391+0000 7ff00659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.391+0000 7ff00659c640 1 -- 172.21.15.49:0/1635728938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00813be10 con 0x7ff008142870 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.394+0000 7ff00659c640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 0x7ff0080ac3c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feffc02bee0 tx=0x7feffc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.394+0000 7fefef7fe640 10 client.?.objecter ms_handle_connect 0x7ff008142870 2024-05-15T12:32:54.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.394+0000 7fefef7fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:54.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 1 -- 172.21.15.49:0/1635728938 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc035040 con 0x7ff008142870 2024-05-15T12:32:54.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:54.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:54.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 10 monclient: dump: 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 10 monclient: _finish_auth 0 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 1 -- 172.21.15.49:0/1635728938 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feffc03b920 con 0x7ff008142870 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7ff00d235640 5 monclient: authenticate success, global_id 4375 2024-05-15T12:32:54.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7ff00d235640 10 monclient: _renew_subs 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7ff00d235640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7ff00d235640 1 -- 172.21.15.49:0/1635728938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0080ad520 con 0x7ff008142870 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 1 -- 172.21.15.49:0/1635728938 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc03bd10 con 0x7ff008142870 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7fefef7fe640 10 monclient: dump: 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.390 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:54.391 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:54.391 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:54.391 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7ff00d235640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:54.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7ff00d235640 10 monclient: _renew_subs 2024-05-15T12:32:54.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7ff00d235640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:54.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7ff00559a640 4 set_mon_vals no callback set 2024-05-15T12:32:54.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7ff00d235640 1 -- 172.21.15.49:0/1635728938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0080adc30 con 0x7ff008142870 2024-05-15T12:32:54.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.398+0000 7ff00d235640 1 librados: init done 2024-05-15T12:32:54.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.399+0000 7ff00d235640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:54.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.399+0000 7ff00d235640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:54.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.399+0000 7ff00d235640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:54.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.399+0000 7ff00d235640 1 -- 172.21.15.49:0/1635728938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00804f0e0 con 0x7ff008142870 2024-05-15T12:32:54.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.651+0000 7fefef7fe640 1 -- 172.21.15.49:0/1635728938 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7feffc038020 con 0x7ff008142870 2024-05-15T12:32:54.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.652+0000 7fefef7fe640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7feff406cdd0 0x7feff406f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.652+0000 7ff006d9d640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7feff406cdd0 0x7feff406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.652+0000 7ff006d9d640 10 monclient: get_auth_request con 0x7feff406cdd0 auth_method 0 2024-05-15T12:32:54.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.652+0000 7ff006d9d640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7feff406cdd0 0x7feff406f290 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feff80011e0 tx=0x7feff8007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:54.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.652+0000 7fefef7fe640 10 client.4375.objecter ms_handle_connect 0x7feff406cdd0 2024-05-15T12:32:54.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.657+0000 7fefef7fe640 1 -- 172.21.15.49:0/1635728938 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feffc0aaec0 con 0x7ff008142870 2024-05-15T12:32:54.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.657+0000 7fefef7fe640 10 client.4375.objecter ms_dispatch 0x7ff008000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:54.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.657+0000 7fefef7fe640 3 client.4375.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:54.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.657+0000 7fefef7fe640 3 client.4375.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:54.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.657+0000 7fefef7fe640 20 client.4375.objecter dump_active .. 0 homeless 2024-05-15T12:32:54.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.667+0000 7fefef7fe640 1 -- 172.21.15.49:0/1635728938 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feffc054020 con 0x7ff008142870 2024-05-15T12:32:54.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.667+0000 7fefef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:54.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.667+0000 7fefef7fe640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:54.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.855+0000 7ff00d235640 1 -- 172.21.15.49:0/1635728938 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff00813b730 con 0x7feff406cdd0 2024-05-15T12:32:54.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.856+0000 7fefef7fe640 1 -- 172.21.15.49:0/1635728938 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+35488 (secure 0 0 0) 0x7ff00813b730 con 0x7feff406cdd0 2024-05-15T12:32:54.846 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:32:54.847 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-05-15T12:32:54.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 10 librados: watch_flush enter 2024-05-15T12:32:54.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 10 librados: watch_flush exit 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 20 client.4375.objecter shutdown clearing up homeless session... 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 10 client.4375.objecter successfully canceled tick 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 -- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7feff406cdd0 msgr2=0x7feff406f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7feff406cdd0 0x7feff406f290 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feff80011e0 tx=0x7feff8007d30 comp rx=0 tx=0).stop 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 10 monclient: shutdown 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 -- 172.21.15.49:0/1635728938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 msgr2=0x7ff0080ac3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 0x7ff0080ac3c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feffc02bee0 tx=0x7feffc0046c0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 -- 172.21.15.49:0/1635728938 shutdown_connections 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0081431f0 0x7ff008133e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7feff406cdd0 0x7feff406f290 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff008142870 0x7ff0080ac3c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 --2- 172.21.15.49:0/1635728938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0080b53e0 0x7ff0080abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 -- 172.21.15.49:0/1635728938 >> 172.21.15.49:0/1635728938 conn(0x7ff008066ed0 msgr2=0x7ff0080a97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:54.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 -- 172.21.15.49:0/1635728938 shutdown_connections 2024-05-15T12:32:54.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 -- 172.21.15.49:0/1635728938 wait complete. 2024-05-15T12:32:54.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.858+0000 7ff00d235640 1 librados: shutdown 2024-05-15T12:32:54.859 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-05-15T12:32:54.287210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218808,"kb_used_data":2624,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755912,"statfs":{"total":773094113280,"available":772870053888,"internally_reserved":0,"allocated":2686976,"data_stored":1387704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.177339"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.314795+0000","last_change":"2024-05-15T12:32:48.258451+0000","last_active":"2024-05-15T12:32:48.314795+0000","last_peered":"2024-05-15T12:32:48.314795+0000","last_clean":"2024-05-15T12:32:48.314795+0000","last_became_active":"2024-05-15T12:32:48.252725+0000","last_became_peered":"2024-05-15T12:32:48.252725+0000","last_unstale":"2024-05-15T12:32:48.314795+0000","last_undegraded":"2024-05-15T12:32:48.314795+0000","last_fullsized":"2024-05-15T12:32:48.314795+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:37:05.959841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:32:49.380976+0000","last_change":"2024-05-15T12:32:48.309674+0000","last_active":"2024-05-15T12:32:49.380976+0000","last_peered":"2024-05-15T12:32:49.380976+0000","last_clean":"2024-05-15T12:32:49.380976+0000","last_became_active":"2024-05-15T12:32:48.283635+0000","last_became_peered":"2024-05-15T12:32:48.283635+0000","last_unstale":"2024-05-15T12:32:49.380976+0000","last_undegraded":"2024-05-15T12:32:49.380976+0000","last_fullsized":"2024-05-15T12:32:49.380976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:07:32.002393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.337225+0000","last_change":"2024-05-15T12:32:47.531833+0000","last_active":"2024-05-15T12:32:48.337225+0000","last_peered":"2024-05-15T12:32:48.337225+0000","last_clean":"2024-05-15T12:32:48.337225+0000","last_became_active":"2024-05-15T12:32:47.526275+0000","last_became_peered":"2024-05-15T12:32:47.526275+0000","last_unstale":"2024-05-15T12:32:48.337225+0000","last_undegraded":"2024-05-15T12:32:48.337225+0000","last_fullsized":"2024-05-15T12:32:48.337225+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:21:07.065217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:32:49.302979+0000","last_change":"2024-05-15T12:32:47.687489+0000","last_active":"2024-05-15T12:32:49.302979+0000","last_peered":"2024-05-15T12:32:49.302979+0000","last_clean":"2024-05-15T12:32:49.302979+0000","last_became_active":"2024-05-15T12:32:47.680498+0000","last_became_peered":"2024-05-15T12:32:47.680498+0000","last_unstale":"2024-05-15T12:32:49.302979+0000","last_undegraded":"2024-05-15T12:32:49.302979+0000","last_fullsized":"2024-05-15T12:32:49.302979+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:09:36.650810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.322083+0000","last_change":"2024-05-15T12:32:47.680657+0000","last_active":"2024-05-15T12:32:48.322083+0000","last_peered":"2024-05-15T12:32:48.322083+0000","last_clean":"2024-05-15T12:32:48.322083+0000","last_became_active":"2024-05-15T12:32:47.670226+0000","last_became_peered":"2024-05-15T12:32:47.670226+0000","last_unstale":"2024-05-15T12:32:48.322083+0000","last_undegraded":"2024-05-15T12:32:48.322083+0000","last_fullsized":"2024-05-15T12:32:48.322083+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:37:30.875058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:32:50.317042+0000","last_change":"2024-05-15T12:32:47.463044+0000","last_active":"2024-05-15T12:32:50.317042+0000","last_peered":"2024-05-15T12:32:50.317042+0000","last_clean":"2024-05-15T12:32:50.317042+0000","last_became_active":"2024-05-15T12:32:47.432679+0000","last_became_peered":"2024-05-15T12:32:47.432679+0000","last_unstale":"2024-05-15T12:32:50.317042+0000","last_undegraded":"2024-05-15T12:32:50.317042+0000","last_fullsized":"2024-05-15T12:32:50.317042+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:13:56.912246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.325652+0000","last_change":"2024-05-15T12:32:47.459103+0000","last_active":"2024-05-15T12:32:48.325652+0000","last_peered":"2024-05-15T12:32:48.325652+0000","last_clean":"2024-05-15T12:32:48.325652+0000","last_became_active":"2024-05-15T12:32:47.430194+0000","last_became_peered":"2024-05-15T12:32:47.430194+0000","last_unstale":"2024-05-15T12:32:48.325652+0000","last_undegraded":"2024-05-15T12:32:48.325652+0000","last_fullsized":"2024-05-15T12:32:48.325652+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:30:01.614452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-15T12:32:48.331314+0000","last_change":"2024-05-15T12:32:47.722217+0000","last_active":"2024-05-15T12:32:48.331314+0000","last_peered":"2024-05-15T12:32:48.331314+0000","last_clean":"2024-05-15T12:32:48.331314+0000","last_became_active":"2024-05-15T12:32:47.716674+0000","last_became_peered":"2024-05-15T12:32:47.716674+0000","last_unstale":"2024-05-15T12:32:48.331314+0000","last_undegraded":"2024-05-15T12:32:48.331314+0000","last_fullsized":"2024-05-15T12:32:48.331314+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:38:58.704500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'50","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-05-15T12:32:50.308588+0000","last_change":"2024-05-15T12:32:43.562093+0000","last_active":"2024-05-15T12:32:50.308588+0000","last_peered":"2024-05-15T12:32:50.308588+0000","last_clean":"2024-05-15T12:32:50.308588+0000","last_became_active":"2024-05-15T12:32:42.889644+0000","last_became_peered":"2024-05-15T12:32:42.889644+0000","last_unstale":"2024-05-15T12:32:50.308588+0000","last_undegraded":"2024-05-15T12:32:50.308588+0000","last_fullsized":"2024-05-15T12:32:50.308588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:30.812143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:30.812143+0000","last_clean_scrub_stamp":"2024-05-15T12:32:30.812143+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:45:22.988060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":672800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":593920,"data_stored":427200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":19,"seq":81604378635,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":90800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378633,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":593920,"data_stored":427200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444041,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":90800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444041,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":90800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":85052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":85052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":90800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:32:54.860 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-15T12:32:54.860 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-15T12:32:54.860 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-15T12:32:54.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-15T12:32:54.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.973+0000 7f7521445640 10 monclient: get_monmap_and_config 2024-05-15T12:32:54.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.973+0000 7f7521445640 10 monclient: build_initial_monmap 2024-05-15T12:32:54.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.973+0000 7f7521445640 10 monclient: monmap: 2024-05-15T12:32:54.963 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:54.964 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:54.964 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:32:54.975140+0000 2024-05-15T12:32:54.964 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:32:54.975140+0000 2024-05-15T12:32:54.964 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:54.964 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.964 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T12:32:54.964 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T12:32:54.964 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T12:32:54.964 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T12:32:54.964 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T12:32:54.965 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.974+0000 7f7521445640 1 Processor -- start 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.974+0000 7f7521445640 1 -- start start 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.974+0000 7f7521445640 10 monclient: init 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.974+0000 7f7521445640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.974+0000 7f7521445640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f751c0b5470 0x7f751c13fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 10 monclient(hunting): picked mon.noname-c con 0x7f751c0b5470 addr v2:172.21.15.88:3300/0 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c0a98c0 0x7f751c145c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 10 monclient(hunting): picked mon.noname-b con 0x7f751c0a98c0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f751c140020 addr v1:172.21.15.49:6789/0 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c0ab400 con 0x7f751c140020 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:54.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751c0b5850 con 0x7f751c0b5470 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751c0a9e70 con 0x7f751c0a98c0 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f7521445640 10 monclient(hunting): authenticate will time out at 1533.137550s 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f751a7fc640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c0a98c0 0x7f751c145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f751affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f751c0b5470 0x7f751c13fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f751b7fe640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f751c140020 0x7f751c146140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:38210/0 (socket says 172.21.15.49:38210) 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.975+0000 7f751b7fe640 1 -- 172.21.15.49:0/386068940 learned_addr learned my addr 172.21.15.49:0/386068940 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751a7fc640 10 monclient(hunting): get_auth_request con 0x7f751c0a98c0 auth_method 0 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751affd640 10 monclient(hunting): get_auth_request con 0x7f751c0b5470 auth_method 0 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751affd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:54.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:54.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:54.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:54.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:54.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:54.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:54.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.976+0000 7f751a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:54.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.978+0000 7f751a7fc640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-05-15T12:32:54.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.979+0000 7f751a7fc640 1 -- 172.21.15.49:0/386068940 >> v2:172.21.15.88:3300/0 conn(0x7f751c0b5470 msgr2=0x7f751c13fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:32:54.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.979+0000 7f751a7fc640 1 --2- 172.21.15.49:0/386068940 >> v2:172.21.15.88:3300/0 conn(0x7f751c0b5470 0x7f751c13fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.979+0000 7f751a7fc640 1 -- 172.21.15.49:0/386068940 >> v1:172.21.15.49:6789/0 conn(0x7f751c140020 legacy=0x7f751c146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.979+0000 7f751a7fc640 10 monclient: _finish_hunting 0 2024-05-15T12:32:54.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.979+0000 7f751a7fc640 1 monclient: found mon.noname-b 2024-05-15T12:32:54.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.979+0000 7f751a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:54.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.979+0000 7f751a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T12:32:54.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.979+0000 7f751a7fc640 1 -- 172.21.15.49:0/386068940 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c0ab6e0 con 0x7f751c0a98c0 2024-05-15T12:32:54.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.981+0000 7f751a7fc640 1 --2- 172.21.15.49:0/386068940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c0a98c0 0x7f751c145c00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75100097a0 tx=0x7f751002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69bd6637fc0fe8e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:54.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 1 -- 172.21.15.49:0/386068940 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75100315a0 con 0x7f751c0a98c0 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 10 monclient: dump: 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:54.976 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 10 monclient: _finish_auth 0 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 10 monclient: _check_auth_tickets 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 1 -- 172.21.15.49:0/386068940 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510037070 con 0x7f751c0a98c0 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7521445640 5 monclient: authenticate success, global_id 4319 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7521445640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7521445640 10 monclient: get_monmap_and_config success 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7521445640 4 set_mon_vals no callback set 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 1 -- 172.21.15.49:0/386068940 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510031960 con 0x7f751c0a98c0 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:54.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.985+0000 7f7519ffb640 10 monclient: dump: 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:54.978 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.986+0000 7f7521445640 10 monclient: shutdown 2024-05-15T12:32:54.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.986+0000 7f7521445640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:54.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.986+0000 7f7521445640 1 -- 172.21.15.49:0/386068940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c0a98c0 msgr2=0x7f751c145c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.986+0000 7f7521445640 1 --2- 172.21.15.49:0/386068940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c0a98c0 0x7f751c145c00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75100097a0 tx=0x7f751002b840 comp rx=0 tx=0).stop 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.986+0000 7f7521445640 1 -- 172.21.15.49:0/386068940 shutdown_connections 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.986+0000 7f7521445640 1 --2- 172.21.15.49:0/386068940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c0a98c0 0x7f751c145c00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.986+0000 7f7521445640 1 --2- 172.21.15.49:0/386068940 >> v2:172.21.15.88:3300/0 conn(0x7f751c0b5470 0x7f751c13fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.986+0000 7f7521445640 1 -- 172.21.15.49:0/386068940 >> 172.21.15.49:0/386068940 conn(0x7f751c066f00 msgr2=0x7f751c067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.986+0000 7f7521445640 1 -- 172.21.15.49:0/386068940 shutdown_connections 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.986+0000 7f7521445640 1 -- 172.21.15.49:0/386068940 wait complete. 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.987+0000 7f7521445640 10 monclient: build_initial_monmap 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.987+0000 7f7521445640 10 monclient: monmap: 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T12:32:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.987+0000 7f7521445640 1 librados: starting msgr at 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.987+0000 7f7521445640 1 librados: starting objecter 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 1 Processor -- start 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 1 -- start start 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 1 librados: setting wanted keys 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 1 librados: calling monclient init 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient: init 2024-05-15T12:32:54.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient: _reopen_session rank -1 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c0a98c0 0x7f751c1db330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient(hunting): picked mon.noname-b con 0x7f751c0a98c0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f751c0b5470 0x7f751c147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient(hunting): picked mon.noname-a con 0x7f751c0b5470 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c140020 0x7f751c1484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient(hunting): picked mon.noname-c con 0x7f751c140020 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c146850 con 0x7f751c0b5470 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c0ab120 con 0x7f751c0a98c0 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient(hunting): start opening mon connection 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751c14d140 con 0x7f751c140020 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient(hunting): _renew_subs 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.988+0000 7f7521445640 10 monclient(hunting): authenticate will time out at 1533.151187s 2024-05-15T12:32:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751b7fe640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c140020 0x7f751c1484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c0a98c0 0x7f751c1db330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f751c0b5470 0x7f751c147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c0a98c0 0x7f751c1db330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:36776/0 (socket says 172.21.15.49:36776) 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751affd640 1 -- 172.21.15.49:0/4290129132 learned_addr learned my addr 172.21.15.49:0/4290129132 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751affd640 10 monclient(hunting): get_auth_request con 0x7f751c0a98c0 auth_method 0 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751affd640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751b7fe640 10 monclient(hunting): get_auth_request con 0x7f751c140020 auth_method 0 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751a7fc640 10 monclient(hunting): get_auth_request con 0x7f751c0b5470 auth_method 0 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:32:54.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:32:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.989+0000 7f751a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:32:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.990+0000 7f751affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.990+0000 7f751affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.990+0000 7f751affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.990+0000 7f751b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.990+0000 7f751b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.990+0000 7f751b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.990+0000 7f751a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:32:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.990+0000 7f751a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:32:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.990+0000 7f751a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:32:54.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.992+0000 7f751affd640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2024-05-15T12:32:54.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.992+0000 7f751affd640 1 -- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c140020 msgr2=0x7f751c1484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.993+0000 7f751affd640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c140020 0x7f751c1484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.993+0000 7f751affd640 1 -- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f751c0b5470 msgr2=0x7f751c147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:54.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.993+0000 7f751affd640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f751c0b5470 0x7f751c147f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:54.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.993+0000 7f751affd640 10 monclient: _finish_hunting 0 2024-05-15T12:32:54.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.993+0000 7f751affd640 1 monclient: found mon.noname-b 2024-05-15T12:32:54.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.993+0000 7f751affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:32:54.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.993+0000 7f751affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:32:54.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.993+0000 7f751affd640 1 -- 172.21.15.49:0/4290129132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c14c670 con 0x7f751c0a98c0 2024-05-15T12:32:54.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.995+0000 7f751affd640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c0a98c0 0x7f751c1db330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f750c00e9e0 tx=0x7f750c00eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:54.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.995+0000 7f75037fe640 10 client.?.objecter ms_handle_connect 0x7f751c0a98c0 2024-05-15T12:32:54.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.995+0000 7f75037fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:32:54.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 1 -- 172.21.15.49:0/4290129132 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c017070 con 0x7f751c0a98c0 2024-05-15T12:32:54.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:54.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 10 monclient: dump: 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 10 monclient: _finish_auth 0 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 10 monclient: _check_auth_tickets 2024-05-15T12:32:54.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 1 -- 172.21.15.49:0/4290129132 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750c00c9d0 con 0x7f751c0a98c0 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 1 -- 172.21.15.49:0/4290129132 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c00cdc0 con 0x7f751c0a98c0 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75037fe640 10 monclient: dump: 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:32:54.991 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f7521445640 5 monclient: authenticate success, global_id 4384 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f7521445640 10 monclient: _renew_subs 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f7521445640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f7521445640 1 -- 172.21.15.49:0/4290129132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751c14c8e0 con 0x7f751c0a98c0 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f75197fa640 4 set_mon_vals no callback set 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f7521445640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f7521445640 10 monclient: _renew_subs 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f7521445640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f7521445640 1 -- 172.21.15.49:0/4290129132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751c1db870 con 0x7f751c0a98c0 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.998+0000 7f7521445640 1 librados: init done 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.999+0000 7f7521445640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.999+0000 7f7521445640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:54.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.999+0000 7f7521445640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:54.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:54.999+0000 7f7521445640 1 -- 172.21.15.49:0/4290129132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751c04f0e0 con 0x7f751c0a98c0 2024-05-15T12:32:55.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.240+0000 7f75037fe640 1 -- 172.21.15.49:0/4290129132 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f750c01a070 con 0x7f751c0a98c0 2024-05-15T12:32:55.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.241+0000 7f75037fe640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f750806ce20 0x7f750806f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:32:55.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.241+0000 7f751a7fc640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f750806ce20 0x7f750806f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:32:55.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.241+0000 7f751a7fc640 10 monclient: get_auth_request con 0x7f750806ce20 auth_method 0 2024-05-15T12:32:55.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.241+0000 7f751a7fc640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f750806ce20 0x7f750806f2e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7510036000 tx=0x7f751003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:32:55.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.241+0000 7f75037fe640 10 client.4384.objecter ms_handle_connect 0x7f750806ce20 2024-05-15T12:32:55.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.244+0000 7f75037fe640 1 -- 172.21.15.49:0/4290129132 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f750c0571e0 con 0x7f751c0a98c0 2024-05-15T12:32:55.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.244+0000 7f75037fe640 10 client.4384.objecter ms_dispatch 0x7f751c000b90 osd_map(26..26 src has 1..26) v4 2024-05-15T12:32:55.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.244+0000 7f75037fe640 3 client.4384.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:32:55.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.244+0000 7f75037fe640 3 client.4384.objecter handle_osd_map decoding full epoch 26 2024-05-15T12:32:55.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.244+0000 7f75037fe640 20 client.4384.objecter dump_active .. 0 homeless 2024-05-15T12:32:55.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.254+0000 7f75037fe640 1 -- 172.21.15.49:0/4290129132 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f750c059140 con 0x7f751c0a98c0 2024-05-15T12:32:55.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.254+0000 7f75037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T12:32:55.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.254+0000 7f75037fe640 10 monclient: _finish_command 1 = system:0 2024-05-15T12:32:55.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.480+0000 7f7521445640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-05-15T12:32:55.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.480+0000 7f7521445640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-05-15T12:32:55.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.480+0000 7f7521445640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:32:55.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.480+0000 7f7521445640 1 -- 172.21.15.49:0/4290129132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f751c0716d0 con 0x7f751c0a98c0 2024-05-15T12:32:55.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.490+0000 7f75037fe640 1 -- 172.21.15.49:0/4290129132 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f750c05d1e0 con 0x7f751c0a98c0 2024-05-15T12:32:55.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.490+0000 7f75037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-05-15T12:32:55.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.490+0000 7f75037fe640 10 monclient: _finish_command 2 = system:0 2024-05-15T12:32:55.481 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T12:32:55.481 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-15T12:32:55.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.492+0000 7f7521445640 10 librados: watch_flush enter 2024-05-15T12:32:55.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.492+0000 7f7521445640 10 librados: watch_flush exit 2024-05-15T12:32:55.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.492+0000 7f7521445640 20 client.4384.objecter shutdown clearing up homeless session... 2024-05-15T12:32:55.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 10 client.4384.objecter successfully canceled tick 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 -- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f750806ce20 msgr2=0x7f750806f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f750806ce20 0x7f750806f2e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7510036000 tx=0x7f751003d040 comp rx=0 tx=0).stop 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 10 monclient: shutdown 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 -- 172.21.15.49:0/4290129132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c0a98c0 msgr2=0x7f751c1db330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c0a98c0 0x7f751c1db330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f750c00e9e0 tx=0x7f750c00eea0 comp rx=0 tx=0).stop 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 -- 172.21.15.49:0/4290129132 shutdown_connections 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f751c140020 0x7f751c1484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f751c0b5470 0x7f751c147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f750806ce20 0x7f750806f2e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 --2- 172.21.15.49:0/4290129132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c0a98c0 0x7f751c1db330 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 -- 172.21.15.49:0/4290129132 >> 172.21.15.49:0/4290129132 conn(0x7f751c066f00 msgr2=0x7f751c0b69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 -- 172.21.15.49:0/4290129132 shutdown_connections 2024-05-15T12:32:55.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 -- 172.21.15.49:0/4290129132 wait complete. 2024-05-15T12:32:55.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T12:32:55.493+0000 7f7521445640 1 librados: shutdown 2024-05-15T12:32:55.494 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-15T12:32:55.495 INFO:teuthology.run_tasks:Running task workunit... 2024-05-15T12:32:55.506 INFO:tasks.workunit:Pulling workunits from ref ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:32:55.507 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-15T12:32:55.507 DEBUG:teuthology.orchestra.run.smithi049:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-15T12:32:55.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:32:55.523 INFO:teuthology.orchestra.run.smithi049.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-15T12:32:55.523 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-05-15T12:32:55.579 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-05-15T12:32:55.579 DEBUG:teuthology.orchestra.run.smithi049:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-05-15T12:32:55.635 INFO:tasks.workunit:timeout=6h 2024-05-15T12:32:55.635 INFO:tasks.workunit:cleanup=True 2024-05-15T12:32:55.635 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:32:55.692 INFO:tasks.workunit.client.0.smithi049.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-15T12:33:00.320 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:01:29.530 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:00.336 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:01:29.546 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:00.371 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:01:29.539 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:00.372 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:01:29.540 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:00.376 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:01:29.586 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:00.379 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:01:29.589 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:00.399 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:01:29.608 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:00.422 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:01:29.590 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:00.437 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:01:29.647 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:20.370 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:01:49.579 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:20.371 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:01:49.581 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:20.386 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:01:49.595 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:20.423 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:01:49.590 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:20.424 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:01:49.592 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:20.424 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:01:49.633 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:20.425 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:01:49.635 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:20.444 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:01:49.611 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:20.447 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:01:49.656 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:40.417 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:02:09.626 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:40.418 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:02:09.628 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:40.443 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:02:09.653 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:40.467 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:02:09.635 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:40.474 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:02:09.642 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:40.490 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:02:09.657 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:40.494 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:02:09.704 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:40.497 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:02:09.707 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:33:40.515 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:02:09.724 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:00.781 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:02:29.990 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:00.786 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:02:29.995 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:00.787 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:02:29.996 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:00.788 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:02:29.997 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:00.805 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:02:30.014 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:00.836 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:02:30.004 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:00.838 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:02:30.006 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:00.851 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:02:30.019 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:00.898 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:02:30.108 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:20.726 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:02:49.936 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:20.779 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:02:49.988 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:20.799 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:02:50.008 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:20.826 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:02:49.994 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:20.827 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:02:49.995 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:20.840 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:02:50.008 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:20.851 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:02:50.060 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:20.853 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:02:50.062 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:20.869 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:02:50.079 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:34:26.441 INFO:tasks.workunit.client.0.smithi049.stderr:Updating files: 93% (11425/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-15T12:34:26.968 INFO:tasks.workunit.client.0.smithi049.stderr:Note: switching to 'ae2481253f9b2fa3451954e615eedb67ebdd5314'. 2024-05-15T12:34:26.968 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-05-15T12:34:26.968 INFO:tasks.workunit.client.0.smithi049.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-15T12:34:26.968 INFO:tasks.workunit.client.0.smithi049.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-15T12:34:26.968 INFO:tasks.workunit.client.0.smithi049.stderr:state without impacting any branches by switching back to a branch. 2024-05-15T12:34:26.968 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-05-15T12:34:26.968 INFO:tasks.workunit.client.0.smithi049.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr: git switch -c 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr:Or undo this operation with: 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr: git switch - 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-05-15T12:34:26.969 INFO:tasks.workunit.client.0.smithi049.stderr:HEAD is now at ae2481253f9 DNM: teuto retrigger 2024-05-15T12:34:26.972 DEBUG:teuthology.orchestra.run.smithi049:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-15T12:34:27.034 INFO:tasks.workunit.client.0.smithi049.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-15T12:34:27.036 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-15T12:34:27.036 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-15T12:34:27.648 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-15T12:34:27.708 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-15T12:34:27.750 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-15T12:34:27.752 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-15T12:34:27.752 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-15T12:34:27.792 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-15T12:34:27.796 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-05-15T12:34:27.796 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-15T12:34:27.854 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-05-15T12:34:27.855 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-05-15T12:34:27.855 DEBUG:teuthology.orchestra.run.smithi049:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae2481253f9b2fa3451954e615eedb67ebdd5314 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-05-15T12:34:27.915 INFO:tasks.workunit.client.0.smithi049.stderr:+ parallel=1 2024-05-15T12:34:27.915 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' '' = --serial ']' 2024-05-15T12:34:27.915 INFO:tasks.workunit.client.0.smithi049.stderr:+ crimson=0 2024-05-15T12:34:27.915 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' '' = --crimson ']' 2024-05-15T12:34:27.915 INFO:tasks.workunit.client.0.smithi049.stderr:+ color= 2024-05-15T12:34:27.915 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -t 1 ']' 2024-05-15T12:34:27.915 INFO:tasks.workunit.client.0.smithi049.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-05-15T12:34:27.915 INFO:tasks.workunit.client.0.smithi049.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-05-15T12:34:27.915 INFO:tasks.workunit.client.0.smithi049.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-05-15T12:34:27.916 INFO:tasks.workunit.client.0.smithi049.stderr:+ declare -A pids 2024-05-15T12:34:27.917 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:27.917 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:27.917 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_aio 2024-05-15T12:34:27.917 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_aio' 2024-05-15T12:34:27.917 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_aio 2024-05-15T12:34:27.917 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:27.918 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_aio 2024-05-15T12:34:27.918 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39898 2024-05-15T12:34:27.918 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_aio on pid 39898' 2024-05-15T12:34:27.918 INFO:tasks.workunit.client.0.smithi049.stdout:test api_aio on pid 39898 2024-05-15T12:34:27.922 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=39898 2024-05-15T12:34:27.922 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:27.922 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:27.922 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-05-15T12:34:27.922 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_aio_pp 2024-05-15T12:34:27.922 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_aio_pp' 2024-05-15T12:34:27.922 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_aio_pp 2024-05-15T12:34:27.922 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:27.922 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:27.923 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_aio_pp 2024-05-15T12:34:27.924 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39905 2024-05-15T12:34:27.925 INFO:tasks.workunit.client.0.smithi049.stdout:test api_aio_pp on pid 39905 2024-05-15T12:34:27.929 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_aio_pp on pid 39905' 2024-05-15T12:34:27.929 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=39905 2024-05-15T12:34:27.929 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_io 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_io' 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_io 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:27.930 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.931 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.932 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.933 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.934 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:27.934 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.934 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:27.934 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.934 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_io 2024-05-15T12:34:27.934 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39914 2024-05-15T12:34:27.934 INFO:tasks.workunit.client.0.smithi049.stdout:test api_io on pid 39914 2024-05-15T12:34:27.939 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_io on pid 39914' 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=39914 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_io_pp 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_io_pp' 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_io_pp 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:27.940 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:27.941 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.942 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:27.943 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_io_pp 2024-05-15T12:34:27.944 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39925 2024-05-15T12:34:27.945 INFO:tasks.workunit.client.0.smithi049.stdout:test api_io_pp on pid 39925 2024-05-15T12:34:27.950 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_io_pp on pid 39925' 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=39925 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_asio 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_asio' 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_asio 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:27.951 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:27.952 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:27.953 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.954 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:27.955 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.956 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_asio 2024-05-15T12:34:27.957 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39939 2024-05-15T12:34:27.958 INFO:tasks.workunit.client.0.smithi049.stdout:test api_asio on pid 39939 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_asio on pid 39939' 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=39939 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_list 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_list' 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_list 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:27.965 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:27.966 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:27.967 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:27.968 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.969 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:27.970 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.971 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_list 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:27.972 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39953 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_list on pid 39953' 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-05-15T12:34:27.973 INFO:tasks.workunit.client.0.smithi049.stdout:test api_list on pid 39953 2024-05-15T12:34:27.981 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=39953 2024-05-15T12:34:27.981 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_lock 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/39898/exe 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_lock' 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_lock 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:27.982 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:27.983 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:27.984 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:27.985 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:27.986 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:27.987 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:27.988 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_aio: /' 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_lock 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:27.989 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_aio.log 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39973 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_lock on pid 39973' 2024-05-15T12:34:27.990 INFO:tasks.workunit.client.0.smithi049.stdout:test api_lock on pid 39973 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=39973 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_lock_pp 2024-05-15T12:34:28.001 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_lock_pp' 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/39905/exe 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_lock_pp 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.002 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.003 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.004 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.005 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.006 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.007 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.008 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.009 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.010 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.010 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.010 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.010 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_lock_pp 2024-05-15T12:34:28.010 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.010 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_aio_pp: /' 2024-05-15T12:34:28.010 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.010 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39997 2024-05-15T12:34:28.010 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_lock_pp on pid 39997' 2024-05-15T12:34:28.010 INFO:tasks.workunit.client.0.smithi049.stdout:test api_lock_pp on pid 39997 2024-05-15T12:34:28.020 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=39997 2024-05-15T12:34:28.020 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.020 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.020 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.020 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-05-15T12:34:28.020 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_misc 2024-05-15T12:34:28.020 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_misc' 2024-05-15T12:34:28.020 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/39914/exe 2024-05-15T12:34:28.020 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.020 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_misc 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.021 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.022 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.023 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.024 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.025 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.026 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.027 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_io.log 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_io: /' 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.028 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_misc 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/39925/exe 2024-05-15T12:34:28.029 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.030 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40018 2024-05-15T12:34:28.030 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.030 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_misc on pid 40018' 2024-05-15T12:34:28.030 INFO:tasks.workunit.client.0.smithi049.stdout:test api_misc on pid 40018 2024-05-15T12:34:28.044 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40018 2024-05-15T12:34:28.044 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.044 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.044 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.044 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.044 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.044 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_misc_pp 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.045 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_misc_pp' 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.046 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.047 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_misc_pp 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.048 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.049 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_io_pp: /' 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.050 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.051 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/39939/exe 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.052 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.053 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.054 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.054 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.054 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.054 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.054 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.054 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.054 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-05-15T12:34:28.054 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.054 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.054 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_misc_pp 2024-05-15T12:34:28.055 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.055 INFO:tasks.workunit.client.0.smithi049.stdout:test api_misc_pp on pid 40039 2024-05-15T12:34:28.067 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40039 2024-05-15T12:34:28.067 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_misc_pp on pid 40039' 2024-05-15T12:34:28.067 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40039 2024-05-15T12:34:28.067 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.067 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.068 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_asio.log 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.069 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_tier_pp 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_tier_pp' 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.070 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_asio: /' 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.071 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.072 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.073 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.074 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.075 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.076 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/39953/exe 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_tier_pp 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.077 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.078 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_tier_pp 2024-05-15T12:34:28.078 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.078 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.078 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.078 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40062 2024-05-15T12:34:28.078 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_tier_pp on pid 40062' 2024-05-15T12:34:28.078 INFO:tasks.workunit.client.0.smithi049.stdout:test api_tier_pp on pid 40062 2024-05-15T12:34:28.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40062 2024-05-15T12:34:28.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.095 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.095 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.095 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_pool 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.096 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_pool' 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_list.log 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_list: /' 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.097 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.098 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.099 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.100 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.101 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.102 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/39973/exe 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.103 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.104 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.105 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_lock.log 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_lock: /' 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.106 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.107 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_pool 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.108 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_pool 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.109 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40096 2024-05-15T12:34:28.110 INFO:tasks.workunit.client.0.smithi049.stdout:test api_pool on pid 40096 2024-05-15T12:34:28.126 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_pool on pid 40096' 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40096 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/39997/exe 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.127 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.128 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.129 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.130 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_snapshots 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.131 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_snapshots' 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.132 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40018/exe 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.133 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_lock_pp: /' 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.134 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.135 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.136 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.137 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.138 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_misc.log 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_snapshots 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_misc: /' 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.139 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.140 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.141 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.141 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_snapshots 2024-05-15T12:34:28.141 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40159 2024-05-15T12:34:28.141 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_snapshots on pid 40159' 2024-05-15T12:34:28.141 INFO:tasks.workunit.client.0.smithi049.stdout:test api_snapshots on pid 40159 2024-05-15T12:34:28.150 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40159 2024-05-15T12:34:28.150 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.150 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.150 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40039/exe 2024-05-15T12:34:28.150 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.150 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.150 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.150 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.150 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.150 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.151 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.152 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.153 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.154 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_misc_pp: /' 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_snapshots_pp 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_snapshots_pp' 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.155 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.156 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.157 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.157 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.157 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.157 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.157 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_snapshots_pp 2024-05-15T12:34:28.157 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.157 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_snapshots_pp 2024-05-15T12:34:28.157 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40222 2024-05-15T12:34:28.157 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_snapshots_pp on pid 40222' 2024-05-15T12:34:28.157 INFO:tasks.workunit.client.0.smithi049.stdout:test api_snapshots_pp on pid 40222 2024-05-15T12:34:28.163 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40222 2024-05-15T12:34:28.163 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.163 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.163 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-05-15T12:34:28.163 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_stat 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_stat' 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.164 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_stat 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.165 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.166 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.167 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40062/exe 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_stat 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40242 2024-05-15T12:34:28.168 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_stat on pid 40242' 2024-05-15T12:34:28.169 INFO:tasks.workunit.client.0.smithi049.stdout:test api_stat on pid 40242 2024-05-15T12:34:28.181 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40242 2024-05-15T12:34:28.181 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.181 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.181 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.181 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.181 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.181 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.182 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.183 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.184 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_stat_pp 2024-05-15T12:34:28.185 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_stat_pp' 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.186 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.187 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40096/exe 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_stat_pp 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_tier_pp: /' 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.188 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.189 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.190 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_pool.log 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_pool: /' 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.191 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_stat_pp 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40159/exe 2024-05-15T12:34:28.192 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40274 2024-05-15T12:34:28.193 INFO:tasks.workunit.client.0.smithi049.stdout:test api_stat_pp on pid 40274 2024-05-15T12:34:28.201 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_stat_pp on pid 40274' 2024-05-15T12:34:28.201 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40274 2024-05-15T12:34:28.201 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.201 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.201 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.201 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.201 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.201 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.202 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.202 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.202 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.202 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.202 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.202 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.202 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.202 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.202 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.202 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_watch_notify 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_watch_notify' 2024-05-15T12:34:28.203 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_snapshots: /' 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.204 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.205 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.206 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_watch_notify 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.207 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.208 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.209 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.209 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.209 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.209 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_watch_notify 2024-05-15T12:34:28.209 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40301 2024-05-15T12:34:28.209 INFO:tasks.workunit.client.0.smithi049.stdout:test api_watch_notify on pid 40301 2024-05-15T12:34:28.220 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_watch_notify on pid 40301' 2024-05-15T12:34:28.220 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40301 2024-05-15T12:34:28.220 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.220 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.220 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.220 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-05-15T12:34:28.220 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.220 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.220 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.221 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_watch_notify_pp 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_watch_notify_pp' 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_watch_notify_pp 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.222 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.223 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.224 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.225 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.226 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.227 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.228 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.228 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40222/exe 2024-05-15T12:34:28.228 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.228 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_watch_notify_pp 2024-05-15T12:34:28.228 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.228 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.228 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40325 2024-05-15T12:34:28.228 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_watch_notify_pp on pid 40325' 2024-05-15T12:34:28.228 INFO:tasks.workunit.client.0.smithi049.stdout:test api_watch_notify_pp on pid 40325 2024-05-15T12:34:28.232 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40325 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.233 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.234 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.235 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.236 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.236 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40242/exe 2024-05-15T12:34:28.236 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.236 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-05-15T12:34:28.236 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [----------] Global test environment set-up. 2024-05-15T12:34:28.236 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [----------] 12 tests from AsioRados 2024-05-15T12:34:28.236 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.111+0000 7f691d037780 10 monclient: get_monmap_and_config 2024-05-15T12:34:28.236 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.111+0000 7f691d037780 10 monclient: build_initial_monmap 2024-05-15T12:34:28.236 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.111+0000 7f691d037780 10 monclient: monmap: 2024-05-15T12:34:28.236 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: epoch 0 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: last_changed 0.000000 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: created 0.000000 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: min_mon_release 0 (unknown) 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: election_strategy: 1 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.111+0000 7f691d037780 1 Processor -- start 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 1 -- start start 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient: init 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient: _reopen_session rank -1 2024-05-15T12:34:28.237 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754006ea0 0x5637540d51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient(hunting): picked mon.noname-b con 0x563754006ea0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563754042820 0x5637541840b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient(hunting): picked mon.noname-c con 0x563754042820 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563754049370 0x5637541845f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient(hunting): picked mon.noname-a con 0x563754049370 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient(hunting): start opening mon connection 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x563754007280 con 0x563754049370 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient(hunting): start opening mon connection 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563753fbab70 con 0x563754006ea0 2024-05-15T12:34:28.238 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient(hunting): start opening mon connection 2024-05-15T12:34:28.239 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x563754042380 con 0x563754042820 2024-05-15T12:34:28.239 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient(hunting): _renew_subs 2024-05-15T12:34:28.239 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.112+0000 7f691d037780 10 monclient(hunting): authenticate will time out at 1626.274355s 2024-05-15T12:34:28.245 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.245 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.245 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.246 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_cmd 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_cmd' 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.247 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.248 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_cmd 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.249 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_cmd 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40363 2024-05-15T12:34:28.250 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_cmd on pid 40363' 2024-05-15T12:34:28.251 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-test api_cmd on pid 40363 2024-05-15T12:34:28.262 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40363 2024-05-15T12:34:28.262 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_stat.log 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.263 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_stat: /' 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.264 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.265 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.266 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_cmd_pp 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_cmd_pp' 2024-05-15T12:34:28.267 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.268 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.269 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.270 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_cmd_pp 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40274/exe 2024-05-15T12:34:28.271 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_cmd_pp 2024-05-15T12:34:28.272 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40405 2024-05-15T12:34:28.272 INFO:tasks.workunit.client.0.smithi049.stdout:test api_cmd_pp on pid 40405 2024-05-15T12:34:28.276 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_cmd_pp on pid 40405' 2024-05-15T12:34:28.276 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40405 2024-05-15T12:34:28.276 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.276 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.276 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.276 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.276 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.276 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.276 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.277 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_service 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.278 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_service' 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stdout:15T12:34:28.122+0000 7f691a28b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754006ea0 0x5637540d51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.126+0000 7f691a28b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754006ea0 0x5637540d51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:38222/0 (socket says 172.21.15.49:38222) 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.126+0000 7f691a28b640 1 -- 172.21.15.49:0/2713844566 learned_addr learned my addr 172.21.15.49:0/2713844566 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.126+0000 7f691a28b640 10 monclient(hunting): get_auth_request con 0x563754006ea0 auth_method 0 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.126+0000 7f691a28b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:34:28.279 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.126+0000 7f691a28b640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.126+0000 7f691a28b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.126+0000 7f691a28b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.126+0000 7f691a28b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.126+0000 7f691a28b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.128+0000 7f691a28b640 10 monclient(hunting): handle_auth_done global_id 4408 payload 274 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.128+0000 7f691a28b640 1 -- 172.21.15.49:0/2713844566 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563754042820 msgr2=0x5637541840b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.128+0000 7f691a28b640 1 --2- 172.21.15.49:0/2713844566 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563754042820 0x5637541840b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.128+0000 7f691a28b640 1 -- 172.21.15.49:0/2713844566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563754049370 msgr2=0x5637541845f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.128+0000 7f691a28b640 1 --2- 172.21.15.49:0/2713844566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563754049370 0x5637541845f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.128+0000 7f691a28b640 10 monclient: _finish_hunting 0 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.128+0000 7f691a28b640 1 monclient: found mon.noname-b 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.128+0000 7f691a28b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:34:28.280 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.128+0000 7f691a28b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:34:28.281 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.128+0000 7f691a28b640 1 -- 172.21.15.49:0/2713844566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637540d4370 con 0x563754006ea0 2024-05-15T12:34:28.281 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.139+0000 7f691a28b640 1 --2- 172.21.15.49:0/2713844566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754006ea0 0x5637540d51a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f690c005c40 tx=0x7f690c04e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=def93b487d64b1a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:34:28.282 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.282 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.282 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.282 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.282 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.282 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.283 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.284 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.284 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.284 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.150+0000 7f6918a88640 1 -- 172.21.15.49:0/2713844566 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c059050 con 0x563754006ea0 2024-05-15T12:34:28.284 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.150+0000 7f6918a88640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:34:28.284 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.150+0000 7f6918a88640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:34:28.284 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.150+0000 7f6918a88640 10 monclient: dump: 2024-05-15T12:34:28.284 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: epoch 1 2024-05-15T12:34:28.284 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:34:28.284 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:34:28.284 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: min_mon_release 19 (squid) 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: election_strategy: 1 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.150+0000 7f6918a88640 10 monclient: _finish_auth 0 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.150+0000 7f6918a88640 10 monclient: _check_auth_tickets 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.150+0000 7f6918a88640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.150+0000 7f6918a88640 1 -- 172.21.15.49:0/2713844566 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690c063070 con 0x563754006ea0 2024-05-15T12:34:28.285 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.150+0000 7f6918a88640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.151+0000 7f6918a88640 1 -- 172.21.15.49:0/2713844566 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c054a30 con 0x563754006ea0 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.151+0000 7f6918a88640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.151+0000 7f6918a88640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.151+0000 7f6918a88640 10 monclient: dump: 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: epoch 1 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: min_mon_release 19 (squid) 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: election_strategy: 1 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:34:28.286 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.287 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.151+0000 7f691d037780 5 monclient: authenticate success, global_id 4408 2024-05-15T12:34:28.287 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.151+0000 7f691d037780 10 monclient: get_monmap_and_config success 2024-05-15T12:34:28.287 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.151+0000 7f691d037780 4 set_mon_vals no callback set 2024-05-15T12:34:28.287 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.152+0000 7f691d037780 10 monclient: shutdown 2024-05-15T12:34:28.287 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.152+0000 7f691d037780 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:34:28.287 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.152+0000 7f691d037780 1 -- 172.21.15.49:0/2713844566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754006ea0 msgr2=0x5637540d51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:34:28.287 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.152+0000 7f691d037780 1 --2- 172.21.15.49:0/2713844566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754006ea0 0x5637540d51a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f690c005c40 tx=0x7f690c04e000 comp rx=0 tx=0).stop 2024-05-15T12:34:28.289 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.289 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.289 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.289 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.289 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.290 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.153+0000 7f691d037780 1 -- 172.21.15.49:0/2713844566 shutdown_connections 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.153+0000 7f691d037780 1 --2- 172.21.15.49:0/2713844566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563754049370 0x5637541845f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.153+0000 7f691d037780 1 --2- 172.21.15.49:0/2713844566 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563754042820 0x5637541840b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.153+0000 7f691d037780 1 --2- 172.21.15.49:0/2713844566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754006ea0 0x5637540d51a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.153+0000 7f691d037780 1 -- 172.21.15.49:0/2713844566 >> 172.21.15.49:0/2713844566 conn(0x563754006110 msgr2=0x563754048050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.154+0000 7f691d037780 1 -- 172.21.15.49:0/2713844566 shutdown_connections 2024-05-15T12:34:28.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.154+0000 7f691d037780 1 -- 172.21.15.49:0/2713844566 wait complete. 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.154+0000 7f691d037780 10 monclient: build_initial_monmap 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.154+0000 7f691d037780 10 monclient: monmap: 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: epoch 0 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: last_changed 0.000000 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: created 0.000000 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: min_mon_release 0 (unknown) 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: election_strategy: 1 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.154+0000 7f691d037780 1 librados: starting msgr at 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.154+0000 7f691d037780 1 librados: starting objecter 2024-05-15T12:34:28.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.154+0000 7f691d037780 1 Processor -- start 2024-05-15T12:34:28.293 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 1 -- start start 2024-05-15T12:34:28.293 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 1 librados: setting wanted keys 2024-05-15T12:34:28.293 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 1 librados: calling monclient init 2024-05-15T12:34:28.293 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient: init 2024-05-15T12:34:28.293 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient: _reopen_session rank -1 2024-05-15T12:34:28.293 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T12:34:28.293 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563754006ea0 0x563754021d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:34:28.293 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient(hunting): picked mon.noname-a con 0x563754006ea0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T12:34:28.293 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754042820 0x563754022280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:34:28.293 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient(hunting): picked mon.noname-b con 0x563754042820 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T12:34:28.295 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40301/exe 2024-05-15T12:34:28.295 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_service 2024-05-15T12:34:28.295 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_stat_pp: /' 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.296 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_service 2024-05-15T12:34:28.297 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.297 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.297 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40431 2024-05-15T12:34:28.297 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_service on pid 40431' 2024-05-15T12:34:28.297 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563754049370 0x5637540d6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 cryptest api_service on pid 40431 2024-05-15T12:34:28.308 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40431 2024-05-15T12:34:28.308 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.308 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.308 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.308 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.308 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.308 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.309 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_service_pp 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.310 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_service_pp' 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.311 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.312 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.313 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40363/exe 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.314 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.315 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.316 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_watch_notify: /' 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_service_pp 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.317 INFO:tasks.workunit.client.0.smithi049.stdout:to rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient(hunting): picked mon.noname-c con 0x563754049370 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient(hunting): start opening mon connection 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637540c84d0 con 0x563754006ea0 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient(hunting): start opening mon connection 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637540c8650 con 0x563754042820 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient(hunting): start opening mon connection 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x5637540d3640 con 0x563754049370 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient(hunting): _renew_subs 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.155+0000 7f691d037780 10 monclient(hunting): authenticate will time out at 1626.317305s 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.163+0000 7f6919a8a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754042820 0x563754022280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.164+0000 7f6919a8a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754042820 0x563754022280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:38284/0 (socket says 172.21.15.49:38284) 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.164+0000 7f6919a8a640 1 -- 172.21.15.49:0/265465264 learned_addr learned my addr 172.21.15.49:0/265465264 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.165+0000 7f6919a8a640 10 monclient(hunting): get_auth_request con 0x563754042820 auth_method 0 2024-05-15T12:34:28.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.165+0000 7f6919a8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T12:34:28.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.165+0000 7f6919a8a640 10 monclient(hunting): _init_auth method 2 2024-05-15T12:34:28.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.165+0000 7f6919a8a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T12:34:28.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.166+0000 7f6919a8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T12:34:28.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.166+0000 7f6919a8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T12:34:28.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.166+0000 7f6919a8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T12:34:28.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.173+0000 7f6919a8a640 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2024-05-15T12:34:28.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.173+0000 7f6919a8a640 1 -- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563754049370 msgr2=0x5637540d6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:34:28.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.173+0000 7f6919a8a640 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563754049370 0x5637540d6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:34:28.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.173+0000 7f6919a8a640 1 -- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563754006ea0 msgr2=0x563754021d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:34:28.323 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.323 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.323 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.323 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.323 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.323 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.324 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.325 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_cmd: /' 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_service_pp 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40477 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_service_pp on pid 40477' 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.326 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.327 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.327 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.327 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.327 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.327 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.327 INFO:tasks.workunit.client.0.smithi049.stdout: api_atest api_service_pp on pid 40477 2024-05-15T12:34:28.332 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40477 2024-05-15T12:34:28.332 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.332 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.332 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_cmd.log 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_c_write_operations 2024-05-15T12:34:28.333 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_c_write_operations' 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.334 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_c_write_operations 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.335 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-05-15T12:34:28.336 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.336 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.336 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.336 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.336 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.336 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.336 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_c_write_operations 2024-05-15T12:34:28.336 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40493 2024-05-15T12:34:28.336 INFO:tasks.workunit.client.0.smithi049.stdout:test api_c_write_operations on pid 40493 2024-05-15T12:34:28.341 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_c_write_operations on pid 40493' 2024-05-15T12:34:28.341 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40493 2024-05-15T12:34:28.341 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.341 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.342 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.343 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_c_read_operations 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.344 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40325/exe 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_c_read_operations' 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40405/exe 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.345 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.346 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.346 INFO:tasks.workunit.client.0.smithi049.stdout:sio: 2024-05-15T12:34:28.173+0000 7f6919a8a640 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563754006ea0 0x563754021d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:34:28.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.173+0000 7f6919a8a640 10 monclient: _finish_hunting 0 2024-05-15T12:34:28.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.173+0000 7f6919a8a640 1 monclient: found mon.noname-b 2024-05-15T12:34:28.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.173+0000 7f6919a8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:34:28.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.173+0000 7f6919a8a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T12:34:28.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.173+0000 7f6919a8a640 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637540c1fe0 con 0x563754042820 2024-05-15T12:34:28.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.180+0000 7f690b7fe640 10 client.?.objecter ms_handle_connect 0x563754042820 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.180+0000 7f690b7fe640 10 client.?.objecter resend_mon_ops 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.179+0000 7f6919a8a640 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754042820 0x563754022280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6910008450 tx=0x7f6910008540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910013620 con 0x563754042820 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 10 monclient: dump: 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: epoch 1 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: min_mon_release 19 (squid) 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: election_strategy: 1 2024-05-15T12:34:28.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 10 monclient: _finish_auth 0 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 10 monclient: _check_auth_tickets 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6910018070 con 0x563754042820 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69100139c0 con 0x563754042820 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690b7fe640 10 monclient: dump: 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: epoch 1 2024-05-15T12:34:28.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:34:28.355 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.355 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.355 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.355 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-05-15T12:34:28.355 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.356 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_c_read_operations 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.357 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.358 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.358 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.358 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.358 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.358 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.358 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-05-15T12:34:28.358 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.358 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.358 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.358 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.359 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.360 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_c_read_operations 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40526 2024-05-15T12:34:28.361 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_c_read_operations on pid 40526' 2024-05-15T12:34:28.362 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: ltest api_c_read_operations on pid 40526 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40526 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.370 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.371 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s api_cls_remote_reads 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' api_cls_remote_reads' 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo api_cls_remote_reads 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.372 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.373 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.374 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.375 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=api_cls_remote_reads 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40536 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test api_cls_remote_reads on pid 40536' 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.376 INFO:tasks.workunit.client.0.smithi049.stdout:test api_cls_remote_reads on pid 40536 2024-05-15T12:34:28.391 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.391 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40536 2024-05-15T12:34:28.391 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.391 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.392 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.393 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.394 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40431/exe 2024-05-15T12:34:28.395 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.396 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s list_parallel 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' list_parallel' 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40477/exe 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.397 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.398 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.399 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.400 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo list_parallel 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.401 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_service.log 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.402 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_service: /' 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.403 INFO:tasks.workunit.client.0.smithi049.stdout:ast_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: created 2024-05-15T12:30:59.910572+0000 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: min_mon_release 19 (squid) 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: election_strategy: 1 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f691d037780 5 monclient: authenticate success, global_id 4423 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f691d037780 10 monclient: _renew_subs 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f691d037780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5637540d9860 con 0x563754042820 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f691d037780 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f691d037780 10 monclient: _renew_subs 2024-05-15T12:34:28.404 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f691d037780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5637540227c0 con 0x563754042820 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f691d037780 1 librados: init done 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f691d037780 10 librados: wait_for_osdmap waiting 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.191+0000 7f690bfff640 4 set_mon_vals no callback set 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.226+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6910013b60 con 0x563754042820 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.227+0000 7f690b7fe640 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f68fc06b100 0x7f68fc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.228+0000 7f691a28b640 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f68fc06b100 0x7f68fc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.228+0000 7f691a28b640 10 monclient: get_auth_request con 0x7f68fc06b100 auth_method 0 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.229+0000 7f691a28b640 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f68fc06b100 0x7f68fc06d5c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f690c05f000 tx=0x7f690c008ce0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.229+0000 7f690b7fe640 10 client.4423.objecter ms_handle_connect 0x7f68fc06b100 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.229+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6910055770 con 0x563754042820 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.229+0000 7f690b7fe640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_map(26..26 src has 1..26) v4 2024-05-15T12:34:28.405 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.229+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map got epochs [26,26] > 0 2024-05-15T12:34:28.409 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-05-15T12:34:28.409 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-05-15T12:34:28.409 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_service_pp: /' 2024-05-15T12:34:28.409 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.410 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.411 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.412 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.412 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.412 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.412 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.412 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.412 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.412 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=list_parallel 2024-05-15T12:34:28.412 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40589 2024-05-15T12:34:28.412 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test list_parallel on pid 40589' 2024-05-15T12:34:28.412 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.229+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding futest list_parallel on pid 40589 2024-05-15T12:34:28.424 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40589 2024-05-15T12:34:28.424 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.424 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.424 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.424 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.424 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.424 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-05-15T12:34:28.424 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.425 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.426 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.427 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.428 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s open_pools_parallel 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' open_pools_parallel' 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.429 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo open_pools_parallel 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.430 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.431 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.432 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40493/exe 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.433 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=open_pools_parallel 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.434 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.435 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.435 INFO:tasks.workunit.client.0.smithi049.stdout:test open_pools_parallel on pid 40628 2024-05-15T12:34:28.441 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.441 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40628 2024-05-15T12:34:28.441 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.441 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.441 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test open_pools_parallel on pid 40628' 2024-05-15T12:34:28.441 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-05-15T12:34:28.441 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40628 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.442 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s delete_pools_parallel 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' delete_pools_parallel' 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo delete_pools_parallel 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.443 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.444 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.445 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40536/exe 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.446 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.447 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.447 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.447 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.447 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=delete_pools_parallel 2024-05-15T12:34:28.447 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40640 2024-05-15T12:34:28.447 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test delete_pools_parallel on pid 40640' 2024-05-15T12:34:28.447 INFO:tasks.workunit.client.0.smithi049.stdout:test delete_pools_parallel on pid 40640 2024-05-15T12:34:28.456 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40640 2024-05-15T12:34:28.456 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.456 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.456 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s cls 2024-05-15T12:34:28.456 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' cls' 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.457 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo cls 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.458 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.459 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.460 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=cls 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40656 2024-05-15T12:34:28.461 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test cls on pid 40656' 2024-05-15T12:34:28.462 INFO:tasks.workunit.client.0.smithi049.stdout:test cls on pid 40656 2024-05-15T12:34:28.483 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40656 2024-05-15T12:34:28.483 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.483 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.483 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.484 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.485 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.486 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.487 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.488 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40589/exe 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.489 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.490 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_list_parallel.log 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ list_parallel: /' 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.491 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.492 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.493 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.494 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s cmd 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' cmd' 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40526/exe 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.495 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.496 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-05-15T12:34:28.497 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.498 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.499 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.500 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo cmd 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=cmd 2024-05-15T12:34:28.501 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40741 2024-05-15T12:34:28.502 INFO:tasks.workunit.client.0.smithi049.stdout:test cmd on pid 40741 2024-05-15T12:34:28.509 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test cmd on pid 40741' 2024-05-15T12:34:28.509 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40741 2024-05-15T12:34:28.509 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.509 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.509 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.509 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.509 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.509 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40640/exe 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.510 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s handler_error 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' handler_error' 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-05-15T12:34:28.511 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.512 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.513 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.514 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.515 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.515 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.515 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.515 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo handler_error 2024-05-15T12:34:28.515 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=handler_error 2024-05-15T12:34:28.515 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40803 2024-05-15T12:34:28.515 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-05-15T12:34:28.515 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test handler_error on pid 40803' 2024-05-15T12:34:28.515 INFO:tasks.workunit.client.0.smithi049.stdout:test handler_error on pid 40803 2024-05-15T12:34:28.525 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40803 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s io 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' io' 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.526 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.527 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo io 2024-05-15T12:34:28.528 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.529 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.530 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.531 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.532 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40628/exe 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.533 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=io 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40870 2024-05-15T12:34:28.534 INFO:tasks.workunit.client.0.smithi049.stdout:test io on pid 40870 2024-05-15T12:34:28.539 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test io on pid 40870' 2024-05-15T12:34:28.539 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40870 2024-05-15T12:34:28.539 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s ec_io 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' ec_io' 2024-05-15T12:34:28.540 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40656/exe 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.541 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.542 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo ec_io 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.543 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.544 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.544 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-05-15T12:34:28.544 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.544 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.544 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.544 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=ec_io 2024-05-15T12:34:28.544 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40891 2024-05-15T12:34:28.544 INFO:tasks.workunit.client.0.smithi049.stdout:test ec_io on pid 40891 2024-05-15T12:34:28.557 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test ec_io on pid 40891' 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40891 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s list 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_cls.log 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' list' 2024-05-15T12:34:28.558 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_cls 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.559 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.560 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo list 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.561 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.562 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.563 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ cls: /' 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.564 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.565 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.566 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.567 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40741/exe 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.568 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.569 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=list 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40918 2024-05-15T12:34:28.570 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test list on pid 40918' 2024-05-15T12:34:28.571 INFO:tasks.workunit.client.0.smithi049.stdout:test list on pid 40918 2024-05-15T12:34:28.591 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40918 2024-05-15T12:34:28.591 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.591 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.591 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.591 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-05-15T12:34:28.591 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.591 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s ec_list 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_cmd 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_cmd.log 2024-05-15T12:34:28.592 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' ec_list' 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.593 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.594 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ cmd: /' 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.595 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.596 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.597 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.598 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40803/exe 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.599 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.600 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_handler_error.log 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_handler_error 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo ec_list 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.601 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.602 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.603 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ handler_error: /' 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.604 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40870/exe 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.605 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.606 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_io.log 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.607 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ io: /' 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_io 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.608 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.609 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40918/exe 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=ec_list 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40991 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test ec_list on pid 40991' 2024-05-15T12:34:28.610 INFO:tasks.workunit.client.0.smithi049.stdout:test ec_list on pid 40991 2024-05-15T12:34:28.617 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=40991 2024-05-15T12:34:28.617 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.617 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.618 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ list: /' 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.619 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.620 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s misc 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' misc' 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.621 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40891/exe 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo misc 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.622 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.623 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.623 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=misc 2024-05-15T12:34:28.623 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.623 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41022 2024-05-15T12:34:28.623 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_ec_io 2024-05-15T12:34:28.623 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test misc on pid 41022' 2024-05-15T12:34:28.623 INFO:tasks.workunit.client.0.smithi049.stdout:test misc on pid 41022 2024-05-15T12:34:28.628 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=41022 2024-05-15T12:34:28.628 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.628 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.628 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-05-15T12:34:28.628 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s pool 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' pool' 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo pool 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_list 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_list.log 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.629 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.630 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.631 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_ec_io.log 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ ec_io: /' 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=pool 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41034 2024-05-15T12:34:28.632 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test pool on pid 41034' 2024-05-15T12:34:28.633 INFO:tasks.workunit.client.0.smithi049.stdout:test pool on pid 41034 2024-05-15T12:34:28.638 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=41034 2024-05-15T12:34:28.638 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.638 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.638 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-05-15T12:34:28.638 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.638 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.638 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.638 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.638 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s read_operations 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' read_operations' 2024-05-15T12:34:28.639 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo read_operations 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.640 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.641 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.642 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.642 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.642 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.642 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.642 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=read_operations 2024-05-15T12:34:28.642 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41053 2024-05-15T12:34:28.642 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test read_operations on pid 41053' 2024-05-15T12:34:28.642 INFO:tasks.workunit.client.0.smithi049.stdout:test read_operations on pid 41053 2024-05-15T12:34:28.650 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=41053 2024-05-15T12:34:28.650 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s snapshots 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' snapshots' 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.651 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.652 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.653 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo snapshots 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.654 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.655 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.656 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=snapshots 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41066 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test snapshots on pid 41066' 2024-05-15T12:34:28.657 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.658 INFO:tasks.workunit.client.0.smithi049.stdout:test snapshots on pid 41066 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=41066 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s watch_notify 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.664 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' watch_notify' 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/40991/exe 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo watch_notify 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.665 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_ec_list 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ ec_list: /' 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_ec_list.log 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.666 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.667 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.668 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.668 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.668 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.668 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.668 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=watch_notify 2024-05-15T12:34:28.668 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.668 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.668 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41083 2024-05-15T12:34:28.668 INFO:tasks.workunit.client.0.smithi049.stdout:test watch_notify on pid 41083 2024-05-15T12:34:28.678 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test watch_notify on pid 41083' 2024-05-15T12:34:28.678 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=41083 2024-05-15T12:34:28.678 INFO:tasks.workunit.client.0.smithi049.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-15T12:34:28.678 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.678 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.678 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.678 INFO:tasks.workunit.client.0.smithi049.stderr:++ printf %25s write_operations 2024-05-15T12:34:28.678 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.678 INFO:tasks.workunit.client.0.smithi049.stderr:+ r=' write_operations' 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.679 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.680 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.681 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.682 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:++ echo write_operations 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.683 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.684 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:++ awk '{print $1}' 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.685 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.686 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.687 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.687 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.687 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.687 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.687 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.687 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.687 INFO:tasks.workunit.client.0.smithi049.stderr:+ ff=write_operations 2024-05-15T12:34:28.687 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41117 2024-05-15T12:34:28.687 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'test write_operations on pid 41117' 2024-05-15T12:34:28.687 INFO:tasks.workunit.client.0.smithi049.stdout:test write_operations on pid 41117 2024-05-15T12:34:28.719 INFO:tasks.workunit.client.0.smithi049.stderr:+ pids[$f]=41117 2024-05-15T12:34:28.719 INFO:tasks.workunit.client.0.smithi049.stderr:+ ret=0 2024-05-15T12:34:28.719 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.719 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.719 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.719 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.719 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' 1 -eq 1 ']' 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40062 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40062 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.720 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.721 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.722 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -f /etc/bashrc ']' 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:+ . /etc/bashrc 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -z '' ']' 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:++ BASHRCSOURCED=Y 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:++ shopt -q login_shell 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.723 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:+++ umask 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/41034/exe 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.724 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.725 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' 0022 -eq 0 ']' 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++ SHELL=/bin/bash 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.726 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-15T12:34:28.727 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.728 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/41022/exe 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.729 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_pool.log 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_pool 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.730 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.731 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_misc.log 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ pool: /' 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'grep=grep --color=auto' 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-15T12:34:28.732 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorls.sh 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' '!' -t 0 ']' 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:+++ return 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export S_COLORS=auto 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:+++ S_COLORS=auto 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_misc 2024-05-15T12:34:28.733 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ misc: /' 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.734 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/41053/exe 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.735 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.736 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_read_operations 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_read_operations.log 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-15T12:34:28.737 INFO:tasks.workunit.client.0.smithi049.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ read_operations: /' 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.738 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.739 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/41083/exe 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-15T12:34:28.740 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/gawk.sh 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-15T12:34:28.741 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/lang.sh 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/locale 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.742 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/41066/exe 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ watch_notify: /' 2024-05-15T12:34:28.743 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_watch_notify 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:++++ export LANG=en_US.UTF-8 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:++++ LANG=en_US.UTF-8 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-15T12:34:28.744 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+++ LANG=en_US.UTF-8 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+++ unset LANG_backup config 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n '' ']' 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' dumb = linux ']' 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.745 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/less.sh 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:++ for i in /etc/profile.d/*.sh 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ snapshots: /' 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:++ '[' '' ']' 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:++ . /etc/profile.d/which2.sh 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:+++++ readlink /proc/41117/exe 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_snapshots.log 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_snapshots 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:++++ basename /usr/bin/bash 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_declare='declare -f' 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:+++ which_opt=-f 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export which_declare 2024-05-15T12:34:28.746 INFO:tasks.workunit.client.0.smithi049.stderr:+++ export -f which 2024-05-15T12:34:28.747 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset i 2024-05-15T12:34:28.747 INFO:tasks.workunit.client.0.smithi049.stderr:++ unset -f pathmunge 2024-05-15T12:34:28.747 INFO:tasks.workunit.client.0.smithi049.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-15T12:34:28.747 INFO:tasks.workunit.client.0.smithi049.stderr:+ export PATH 2024-05-15T12:34:28.747 INFO:tasks.workunit.client.0.smithi049.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-15T12:34:28.747 INFO:tasks.workunit.client.0.smithi049.stderr:+ unset rc 2024-05-15T12:34:28.747 INFO:tasks.workunit.client.0.smithi049.stderr:+ ceph_test_neorados_write_operations 2024-05-15T12:34:28.747 INFO:tasks.workunit.client.0.smithi049.stderr:+ tee ceph_test_neorados_write_operations.log 2024-05-15T12:34:28.747 INFO:tasks.workunit.client.0.smithi049.stderr:+ sed 's/^/ write_operations: /' 2024-05-15T12:34:30.821 INFO:tasks.workunit.client.0.smithi049.stdout:ll epoch 26 2024-05-15T12:34:30.821 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.229+0000 7f690b7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-05-15T12:34:30.821 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.229+0000 7f691d037780 10 librados: wait_for_osdmap done waiting 2024-05-15T12:34:30.821 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.229+0000 7f691d037780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:34:30.821 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.229+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f690c063070 con 0x563754042820 2024-05-15T12:34:30.821 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.267+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6910053970 con 0x563754042820 2024-05-15T12:34:30.821 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.267+0000 7f690b7fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.267+0000 7f6918a88640 10 client.4423.objecter _get_latest_version latest 26, have it 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.267+0000 7f691d037780 10 client.4423.objecter create_pool name=ceph_test_rados_api_asio 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.267+0000 7f691d037780 10 client.4423.objecter pool_op_submit 1 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.267+0000 7f691d037780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:28.267+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x563753fc1420 con 0x563754042820 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.729+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v28) v1 ==== 43+0+0 (secure 0 0 0) 0x7f6910057dd0 con 0x563754042820 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.729+0000 7f690b7fe640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 pool_op_reply(tid 1 (0) Success v28) v1 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.729+0000 7f690b7fe640 10 client.4423.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v28) v1 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.729+0000 7f690b7fe640 10 client.4423.objecter have request 1 at 0x5637540d69e0 Op: create 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.729+0000 7f690b7fe640 20 client.4423.objecter waiting for client to reach epoch 28 before calling back 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.729+0000 7f690b7fe640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.729+0000 7f690b7fe640 10 monclient: _renew_subs 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.729+0000 7f690b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:34:30.822 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.729+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=27}) v3 -- 0x7f68fc070e70 con 0x563754042820 2024-05-15T12:34:30.823 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.729+0000 7f690b7fe640 10 client.4423.objecter done 2024-05-15T12:34:30.823 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 8 ==== osd_map(27..28 src has 1..28) v4 ==== 6497+0+0 (secure 0 0 0) 0x7f691006f1d0 con 0x563754042820 2024-05-15T12:34:30.823 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f690b7fe640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_map(27..28 src has 1..28) v4 2024-05-15T12:34:30.823 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map got epochs [27,28] > 26 2024-05-15T12:34:30.823 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 27 2024-05-15T12:34:32.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 28 2024-05-15T12:34:32.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f690b7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 10 librados: set snap read head -> 1 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 10 librados: writefull oid=exist nspace= 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637540c3760 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 28 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x5637540c6b00 0x563754195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 2 osd.2 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 2 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 15 client.4423.objecter _send_op 2 to 7.1 on osd.2 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:2 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) v9 -- 0x5637540d7670 con 0x5637540c6b00 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:32.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:30.832+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:32.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:31.155+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:32.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:31.155+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:32.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:31.155+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:32.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:31.155+0000 7f690a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:34:32.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:31.368+0000 7f691aa8c640 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x5637540c6b00 0x563754195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T12:34:32.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:31.370+0000 7f691aa8c640 10 monclient: get_auth_request con 0x5637540c6b00 auth_method 0 2024-05-15T12:34:32.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:31.373+0000 7f690b7fe640 10 client.4423.objecter ms_handle_connect 0x5637540c6b00 2024-05-15T12:34:32.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:31.373+0000 7f691aa8c640 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x5637540c6b00 0x563754195470 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T12:34:32.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:32.155+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:32.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:32.155+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:32.155+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:33.156+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:33.156+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:33.156+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:33.192+0000 7f691c4ef640 10 client.4423.objecter tick 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:34.156+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:34.156+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:34.156+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:35.163+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:35.163+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:35.163+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:36.163+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:36.163+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:36.163+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:37.163+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:37.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:37.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:38.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:38.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:38.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:38.193+0000 7f691c4ef640 10 client.4423.objecter tick 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:39.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:39.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:39.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:40.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:40.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:40.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:41.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:41.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:41.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:41.164+0000 7f690a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:42.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:42.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:42.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.163+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.163+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.163+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:43.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.193+0000 7f691c4ef640 10 client.4423.objecter tick 2024-05-15T12:34:43.189 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.193+0000 7f691c4ef640 2 client.4423.objecter tid 2 on osd.2 is laggy 2024-05-15T12:34:43.189 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.193+0000 7f691c4ef640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:34:43.189 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.193+0000 7f691c4ef640 10 monclient: _renew_subs 2024-05-15T12:34:43.189 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.193+0000 7f691c4ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.193+0000 7f691c4ef640 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=29}) v3 -- 0x7f68ec002f90 con 0x563754042820 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.193+0000 7f691c4ef640 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- ping magic: 0 v1 -- 0x7f68ec003360 con 0x5637540c6b00 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 9 ==== osd_map(29..36 src has 1..36) v4 ==== 27099+0+0 (secure 0 0 0) 0x7f6910053d40 con 0x563754042820 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_map(29..36 src has 1..36) v4 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map got epochs [29,36] > 28 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 29 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 29 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 30 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 30 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:43.190 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.199+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 31 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 31 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 32 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 32 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 33 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:43.191 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 33 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:48.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 34 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 34 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 35 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 35 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 36 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 36 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-05-15T12:34:48.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:43.200+0000 7f690b7fe640 20 client.4423.objecter 2 7.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:44.163+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:44.163+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:44.163+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:45.163+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:45.163+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:45.163+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:46.163+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:46.163+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:46.163+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:47.163+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:47.163+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:47.163+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.163+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.163+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.163+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:48.185 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.194+0000 7f691c4ef640 10 client.4423.objecter tick 2024-05-15T12:34:48.186 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.194+0000 7f691c4ef640 2 client.4423.objecter tid 2 on osd.2 is laggy 2024-05-15T12:34:48.186 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.194+0000 7f691c4ef640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:34:48.186 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.194+0000 7f691c4ef640 10 monclient: _renew_subs 2024-05-15T12:34:52.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.194+0000 7f691c4ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:34:52.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.194+0000 7f691c4ef640 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=37}) v3 -- 0x7f68ec003560 con 0x563754042820 2024-05-15T12:34:52.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.194+0000 7f691c4ef640 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- ping magic: 0 v1 -- 0x7f68ec0037e0 con 0x5637540c6b00 2024-05-15T12:34:52.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 10 ==== osd_map(37..38 src has 1..38) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f6910057930 con 0x563754042820 2024-05-15T12:34:52.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_map(37..38 src has 1..38) v4 2024-05-15T12:34:52.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map got epochs [37,38] > 36 2024-05-15T12:34:52.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 37 2024-05-15T12:34:52.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:52.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 37 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 38 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 38 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:48.581+0000 7f690b7fe640 20 client.4423.objecter 2 7.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:49.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:49.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:49.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:50.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:52.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:50.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:52.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:50.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:52.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:51.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:52.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:51.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:52.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:51.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:52.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:51.164+0000 7f690a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T12:34:52.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:52.159+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 1 ==== osd_map(29..38 src has 1..38) v4 ==== 29473+0+0 (crc 0 0 0) 0x7f691400e040 con 0x5637540c6b00 2024-05-15T12:34:52.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:52.159+0000 7f690b7fe640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_map(29..38 src has 1..38) v4 2024-05-15T12:34:52.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:52.159+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map ignoring epochs [29,38] <= 38 2024-05-15T12:34:55.113 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:52.159+0000 7f690b7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-05-15T12:34:55.113 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:52.159+0000 7f690b7fe640 20 client.4423.objecter 2 7.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2024-05-15T12:34:55.113 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:52.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:52.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:52.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.195+0000 7f691c4ef640 10 client.4423.objecter tick 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.195+0000 7f691c4ef640 2 client.4423.objecter tid 2 on osd.2 is laggy 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.195+0000 7f691c4ef640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.195+0000 7f691c4ef640 10 monclient: _renew_subs 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.195+0000 7f691c4ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.195+0000 7f691c4ef640 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=39}) v3 -- 0x7f68ec0039e0 con 0x563754042820 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.195+0000 7f691c4ef640 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- ping magic: 0 v1 -- 0x7f68ec003c70 con 0x5637540c6b00 2024-05-15T12:34:55.114 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.597+0000 7f690b7fe640 1 -- 172.21.15.49:0/265465264 <== mon.1 v2:172.21.15.88:3300/0 11 ==== osd_map(39..39 src has 1..39) v4 ==== 1000+0+0 (secure 0 0 0) 0x7f6910057b30 con 0x563754042820 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.597+0000 7f690b7fe640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_map(39..39 src has 1..39) v4 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.597+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map got epochs [39,39] > 38 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.597+0000 7f690b7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 39 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.597+0000 7f690b7fe640 10 client.4423.objecter checking op 2 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.597+0000 7f690b7fe640 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.597+0000 7f690b7fe640 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.597+0000 7f690b7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:53.597+0000 7f690b7fe640 20 client.4423.objecter 2 7.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:54.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:54.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:54.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 2 ==== osd_op_reply(2 exist [writefull 0~5] v38'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(2 exist [writefull 0~5] v38'1 uv1 ondisk = 0) v8 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 2 ondisk uv 1 in 7.1 attempt 0 2024-05-15T12:34:55.115 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 2 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691aa8c640 15 client.4423.objecter _finish_op 2 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 2 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 10 librados: Objecter returned from writefull r=0 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637540c2750 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 3 osd.2 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 3 2024-05-15T12:34:55.116 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 15 client.4423.objecter _send_op 3 to 7.1 on osd.2 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:3 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x5637540c2c00 con 0x5637540c6b00 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637540c3010 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 20 client.4423.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 10 client.4423.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 4 osd.2 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 4 2024-05-15T12:34:55.117 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 15 client.4423.objecter _send_op 4 to 7.1 on osd.2 2024-05-15T12:34:55.378 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:4 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x5637540c34c0 con 0x5637540c6b00 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.123+0000 7f691d037780 5 client.4423.objecter 2 in flight 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.388+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.388+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.388+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.388+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 3 ondisk uv 1 in 7.1 attempt 0 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.388+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 5 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.388+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 3 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.388+0000 7f691aa8c640 15 client.4423.objecter _finish_op 3 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.388+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.388+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 3 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.388+0000 7f691aa8c640 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.379 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 4 ondisk uv 0 in 7.1 attempt 0 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 4 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 15 client.4423.objecter _finish_op 4 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 4 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:55.380 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (265 ms) 2024-05-15T12:34:55.453 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-05-15T12:34:55.482 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637540c3010 2024-05-15T12:34:55.482 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:55.482 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:55.482 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.482 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.482 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 5 osd.2 2024-05-15T12:34:55.482 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.482 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 5 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 15 client.4423.objecter _send_op 5 to 7.1 on osd.2 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:5 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x5637540c2750 con 0x5637540c6b00 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637541959b0 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 20 client.4423.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 10 client.4423.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 6 osd.2 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 6 2024-05-15T12:34:55.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 15 client.4423.objecter _send_op 6 to 7.1 on osd.2 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:6 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x563754195e60 con 0x5637540c6b00 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.389+0000 7f691d037780 5 client.4423.objecter 2 in flight 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 5 ondisk uv 1 in 7.1 attempt 0 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 5 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 5 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 15 client.4423.objecter _finish_op 5 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 5 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:34:55.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 6 ondisk uv 0 in 7.1 attempt 0 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 6 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 15 client.4423.objecter _finish_op 6 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 6 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (75 ms) 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637541959b0 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 7 osd.2 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 7 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 15 client.4423.objecter _send_op 7 to 7.1 on osd.2 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:7 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x5637540c2ff0 con 0x5637540c6b00 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637541d6890 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 20 client.4423.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-05-15T12:34:55.489 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 10 client.4423.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 8 osd.2 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 8 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 15 client.4423.objecter _send_op 8 to 7.1 on osd.2 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:8 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x5637541d6d40 con 0x5637540c6b00 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.463+0000 7f691d037780 5 client.4423.objecter 2 in flight 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-05-15T12:34:55.490 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.491 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 7 ondisk uv 1 in 7.1 attempt 0 2024-05-15T12:34:55.491 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 5 2024-05-15T12:34:55.491 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 7 2024-05-15T12:34:55.491 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 15 client.4423.objecter _finish_op 7 2024-05-15T12:34:55.491 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:55.491 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 7 2024-05-15T12:34:55.491 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.491 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.576 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:34:55.576 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.576 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 8 ondisk uv 0 in 7.1 attempt 0 2024-05-15T12:34:55.576 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:34:55.576 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:55.576 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 8 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 15 client.4423.objecter _finish_op 8 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 8 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (31 ms) 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 20 librados: queue_aio_write 0x563753f7d830 completion 0x7f6900002970 write_seq 1 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637541959b0 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 9 osd.2 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.577 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 9 2024-05-15T12:34:55.578 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 15 client.4423.objecter _send_op 9 to 7.1 on osd.2 2024-05-15T12:34:55.578 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:9 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) v9 -- 0x563754197210 con 0x5637540c6b00 2024-05-15T12:34:55.578 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.578 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.578 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.495+0000 7f691d037780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-05-15T12:34:55.578 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 9 ==== osd_op_reply(9 exist [write 0~5] v39'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.578 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(9 exist [write 0~5] v39'2 uv2 ondisk = 0) v8 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 9 ondisk uv 2 in 7.1 attempt 0 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 9 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 15 client.4423.objecter _finish_op 9 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 9 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691aa8c640 20 librados: complete_aio_write 0x7f6900002970 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (92 ms) 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-05-15T12:34:55.635 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-05-15T12:34:55.636 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 20 librados: queue_aio_write 0x563753f7d830 completion 0x563753f597b0 write_seq 2 2024-05-15T12:34:55.636 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637541959b0 2024-05-15T12:34:55.636 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:55.636 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:55.636 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.636 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.636 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 10 osd.2 2024-05-15T12:34:55.636 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.636 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 10 2024-05-15T12:34:55.637 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 15 client.4423.objecter _send_op 10 to 7.1 on osd.2 2024-05-15T12:34:55.637 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.586+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:10 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) v9 -- 0x563754197210 con 0x5637540c6b00 2024-05-15T12:34:55.637 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.587+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.637 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.587+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.637 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.587+0000 7f691d037780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-05-15T12:34:55.637 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 10 ==== osd_op_reply(10 exist [write 0~5] v39'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.637 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(10 exist [write 0~5] v39'3 uv3 ondisk = 0) v8 2024-05-15T12:34:55.637 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.637 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 10 ondisk uv 3 in 7.1 attempt 0 2024-05-15T12:34:55.673 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:55.673 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 10 2024-05-15T12:34:55.673 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 15 client.4423.objecter _finish_op 10 2024-05-15T12:34:55.673 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.673 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 10 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691aa8c640 20 librados: complete_aio_write 0x563753f597b0 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (59 ms) 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 20 librados: queue_aio_write 0x563753f7d830 completion 0x5637541959b0 write_seq 3 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637541b9b10 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 11 osd.2 2024-05-15T12:34:55.674 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 11 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 15 client.4423.objecter _send_op 11 to 7.1 on osd.2 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:11 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) v9 -- 0x5637541b9fc0 con 0x5637540c6b00 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.645+0000 7f691d037780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 11 ==== osd_op_reply(11 exist [write 0~5] v39'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(11 exist [write 0~5] v39'4 uv4 ondisk = 0) v8 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 11 ondisk uv 4 in 7.1 attempt 0 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 11 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 15 client.4423.objecter _finish_op 11 2024-05-15T12:34:55.675 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 11 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691aa8c640 20 librados: complete_aio_write 0x5637541959b0 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (38 ms) 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x563754196200 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 12 osd.2 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 12 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 15 client.4423.objecter _send_op 12 to 7.1 on osd.2 2024-05-15T12:34:55.676 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:12 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x5637541966b0 con 0x5637540c6b00 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x563754196a50 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 20 client.4423.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 10 client.4423.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 13 osd.2 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 13 2024-05-15T12:34:55.677 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 15 client.4423.objecter _send_op 13 to 7.1 on osd.2 2024-05-15T12:34:55.744 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:13 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x5637541ba420 con 0x5637540c6b00 2024-05-15T12:34:55.744 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:55.744 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.683+0000 7f691d037780 5 client.4423.objecter 2 in flight 2024-05-15T12:34:55.744 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.744 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-05-15T12:34:55.744 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.744 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 12 ondisk uv 4 in 7.1 attempt 0 2024-05-15T12:34:55.744 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 5 2024-05-15T12:34:55.744 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 12 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 15 client.4423.objecter _finish_op 12 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 12 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 5 client.4423.objecter 1 in flight 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 13 ondisk uv 0 in 7.1 attempt 0 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 13 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 15 client.4423.objecter _finish_op 13 2024-05-15T12:34:55.745 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:55.746 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 13 2024-05-15T12:34:55.746 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.753+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:55.746 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (70 ms) 2024-05-15T12:34:55.746 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-05-15T12:34:55.746 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x563754196310 2024-05-15T12:34:55.746 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:55.746 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 14 osd.2 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 14 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 15 client.4423.objecter _send_op 14 to 7.1 on osd.2 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:14 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x5637541967c0 con 0x5637540c6b00 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637541ba420 2024-05-15T12:34:56.221 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 20 client.4423.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 10 client.4423.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 15 osd.2 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 15 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 15 client.4423.objecter _send_op 15 to 7.1 on osd.2 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:15 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x563754196d80 con 0x5637540c6b00 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:55.754+0000 7f691d037780 5 client.4423.objecter 2 in flight 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.164+0000 7f690a7fc640 10 monclient: tick 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.164+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.164+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T12:34:56.222 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 14 ondisk uv 4 in 7.1 attempt 0 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 5 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 14 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 15 client.4423.objecter _finish_op 14 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 14 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 5 client.4423.objecter 1 in flight 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:56.223 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 15 ondisk uv 0 in 7.1 attempt 0 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 15 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 15 client.4423.objecter _finish_op 15 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 15 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (477 ms) 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637541b94a0 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:56.224 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:56.225 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:56.225 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:56.225 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 16 osd.2 2024-05-15T12:34:56.225 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:56.225 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 16 2024-05-15T12:34:56.225 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 15 client.4423.objecter _send_op 16 to 7.1 on osd.2 2024-05-15T12:34:56.225 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:16 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x563754196d80 con 0x5637540c6b00 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637541b9a60 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 20 client.4423.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 10 client.4423.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 17 osd.2 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 17 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 15 client.4423.objecter _send_op 17 to 7.1 on osd.2 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:17 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) v9 -- 0x5637541b9f10 con 0x5637540c6b00 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:56.241 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.231+0000 7f691d037780 5 client.4423.objecter 2 in flight 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 16 ondisk uv 4 in 7.1 attempt 0 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 5 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 16 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 15 client.4423.objecter _finish_op 16 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 5 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 16 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 5 client.4423.objecter 1 in flight 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-05-15T12:34:56.242 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:56.335 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 17 ondisk uv 0 in 7.1 attempt 0 2024-05-15T12:34:56.335 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-15T12:34:56.335 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:56.335 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 17 2024-05-15T12:34:56.335 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 15 client.4423.objecter _finish_op 17 2024-05-15T12:34:56.335 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.335 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 17 2024-05-15T12:34:56.335 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:56.335 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (21 ms) 2024-05-15T12:34:56.335 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 20 librados: queue_aio_write 0x563753f7d830 completion 0x563754196c70 write_seq 4 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x563754196310 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 18 osd.2 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 18 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 15 client.4423.objecter _send_op 18 to 7.1 on osd.2 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:18 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) v9 -- 0x563754196d80 con 0x5637540c6b00 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.251+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:56.336 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 18 ==== osd_op_reply(18 exist [writefull 0~5] v39'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:56.337 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(18 exist [writefull 0~5] v39'5 uv5 ondisk = 0) v8 2024-05-15T12:34:56.337 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:56.337 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 18 ondisk uv 5 in 7.1 attempt 0 2024-05-15T12:34:56.337 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:56.337 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 18 2024-05-15T12:34:56.427 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 15 client.4423.objecter _finish_op 18 2024-05-15T12:34:56.427 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.427 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 18 2024-05-15T12:34:56.427 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:56.427 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691aa8c640 20 librados: complete_aio_write 0x563754196c70 2024-05-15T12:34:56.427 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (94 ms) 2024-05-15T12:34:56.427 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-05-15T12:34:56.427 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 20 librados: queue_aio_write 0x563753f7d830 completion 0x563754196200 write_seq 5 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x563754196c70 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 19 osd.2 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 19 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 15 client.4423.objecter _send_op 19 to 7.1 on osd.2 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:19 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) v9 -- 0x563754196310 con 0x5637540c6b00 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.345+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 19 ==== osd_op_reply(19 exist [writefull 0~5] v39'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:56.428 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(19 exist [writefull 0~5] v39'6 uv6 ondisk = 0) v8 2024-05-15T12:34:56.429 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:56.429 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 19 ondisk uv 6 in 7.1 attempt 0 2024-05-15T12:34:56.429 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:56.429 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 19 2024-05-15T12:34:56.429 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 15 client.4423.objecter _finish_op 19 2024-05-15T12:34:56.429 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.429 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 19 2024-05-15T12:34:56.482 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691aa8c640 20 librados: complete_aio_write 0x563754196200 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (92 ms) 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 20 librados: queue_aio_write 0x563753f7d830 completion 0x563754196a90 write_seq 6 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 10 client.4423.objecter _op_submit op 0x5637541b94a0 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 20 client.4423.objecter _calc_target epoch 39 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 20 client.4423.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 10 client.4423.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 20 client.4423.objecter _get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 10 client.4423.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 20 osd.2 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 20 client.4423.objecter get_session s=0x5637540c3c10 osd=2 3 2024-05-15T12:34:56.483 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 15 client.4423.objecter _session_op_assign 2 20 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 15 client.4423.objecter _send_op 20 to 7.1 on osd.2 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- osd_op(unknown.0.0:20 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) v9 -- 0x563754196310 con 0x5637540c6b00 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.437+0000 7f691d037780 5 client.4423.objecter 1 in flight 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 <== osd.2 v2:172.21.15.49:6826/2361777220 20 ==== osd_op_reply(20 exist [writefull 0~5] v39'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f691400a380 con 0x5637540c6b00 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 10 client.4423.objecter ms_dispatch 0x563753fb49d0 osd_op_reply(20 exist [writefull 0~5] v39'7 uv7 ondisk = 0) v8 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 10 client.4423.objecter in handle_osd_op_reply 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 7 client.4423.objecter handle_osd_op_reply 20 ondisk uv 7 in 7.1 attempt 0 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 10 client.4423.objecter op 0 rval 0 len 0 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 15 client.4423.objecter handle_osd_op_reply completed tid 20 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 15 client.4423.objecter _finish_op 20 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 4 2024-05-15T12:34:56.484 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 15 client.4423.objecter _session_op_remove 2 20 2024-05-15T12:34:56.485 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 5 client.4423.objecter 0 in flight 2024-05-15T12:34:56.485 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 20 librados: complete_aio_write 0x563754196a90 2024-05-15T12:34:56.485 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (55 ms) 2024-05-15T12:34:56.485 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 10 librados: watch_flush enter 2024-05-15T12:34:56.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 10 librados: watch_flush exit 2024-05-15T12:34:56.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 10 client.4423.objecter close_session for osd.2 2024-05-15T12:34:56.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x5637540c6b00 msgr2=0x563754195470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:34:56.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x5637540c6b00 0x563754195470 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T12:34:56.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 20 client.4423.objecter put_session s=0x5637540c3c10 osd=2 1 2024-05-15T12:34:56.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 20 client.4423.objecter shutdown clearing up homeless session... 2024-05-15T12:34:56.486 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 10 client.4423.objecter successfully canceled tick 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f68fc06b100 msgr2=0x7f68fc06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f68fc06b100 0x7f68fc06d5c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f690c05f000 tx=0x7f690c008ce0 comp rx=0 tx=0).stop 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 10 monclient: shutdown 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754042820 msgr2=0x563754022280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 --2- 172.21.15.49:0/265465264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563754042820 0x563754022280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6910008450 tx=0x7f6910008540 comp rx=0 tx=0).stop 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691aa8c640 1 -- 172.21.15.49:0/265465264 reap_dead start 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 shutdown_connections 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 >> 172.21.15.49:0/265465264 conn(0x563754006110 msgr2=0x5637540c78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 shutdown_connections 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 -- 172.21.15.49:0/265465264 wait complete. 2024-05-15T12:34:56.487 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.493+0000 7f691d037780 1 librados: shutdown 2024-05-15T12:34:56.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [----------] 12 tests from AsioRados (1369 ms total) 2024-05-15T12:34:56.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: 2024-05-15T12:34:56.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [----------] Global test environment tear-down 2024-05-15T12:34:56.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (28384 ms total) 2024-05-15T12:34:56.488 INFO:tasks.workunit.client.0.smithi049.stdout: api_asio: [ PASSED ] 12 tests. 2024-05-15T12:34:58.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-05-15T12:34:58.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [----------] Global test environment set-up. 2024-05-15T12:34:58.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [----------] 4 tests from LibRadosService 2024-05-15T12:34:58.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-05-15T12:34:58.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5804 ms) 2024-05-15T12:34:58.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-05-15T12:34:58.346 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [ OK ] LibRadosService.RegisterLate (64 ms) 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: cluster: 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: id: 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: health: HEALTH_WARN 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 28 pool(s) do not have an application enabled 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: services: 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: osd: 8 osds: 8 up (since 114s), 8 in (since 2m) 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: data: 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: pools: 38 pools, 1161 pgs 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: objects: 2 objects, 449 KiB 2024-05-15T12:34:58.347 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: pgs: 92.765% pgs unknown 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 6.202% pgs not active 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 1077 unknown 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 72 creating+peering 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 12 active+clean 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: cluster: 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: id: 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: health: HEALTH_WARN 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 17 pool(s) do not have an application enabled 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: services: 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-05-15T12:34:58.348 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: osd: 8 osds: 8 up (since 117s), 8 in (since 2m) 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: data: 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: pools: 36 pools, 1049 pgs 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: objects: 2 objects, 449 KiB 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: pgs: 58.913% pgs unknown 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 40.229% pgs not active 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 618 unknown 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 335 creating+peering 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 87 creating+activating 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 9 active+clean 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 2024-05-15T12:34:58.349 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 2024-05-15T12:34:58.350 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4146 ms) 2024-05-15T12:34:58.350 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [ RUN ] LibRadosService.Status 2024-05-15T12:34:58.350 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [ OK ] LibRadosService.Status (20064 ms) 2024-05-15T12:34:58.350 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [----------] 4 tests from LibRadosService (30078 ms total) 2024-05-15T12:34:58.350 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: 2024-05-15T12:34:58.350 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [----------] Global test environment tear-down 2024-05-15T12:34:58.350 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [==========] 4 tests from 1 test suite ran. (30078 ms total) 2024-05-15T12:34:58.350 INFO:tasks.workunit.client.0.smithi049.stdout: api_service: [ PASSED ] 4 tests. 2024-05-15T12:35:00.720 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-05-15T12:35:00.720 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [----------] Global test environment set-up. 2024-05-15T12:35:00.720 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-05-15T12:35:00.720 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (6021 ms) 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (858 ms) 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20401 ms) 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: attempt 0 of 20 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5181 ms) 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (32461 ms total) 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [----------] Global test environment tear-down 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (32461 ms total) 2024-05-15T12:35:00.721 INFO:tasks.workunit.client.0.smithi049.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-05-15T12:35:11.518 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: Running main() from gmock_main.cc 2024-05-15T12:35:11.519 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-05-15T12:35:11.519 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: [----------] Global test environment set-up. 2024-05-15T12:35:11.519 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-05-15T12:35:11.519 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-05-15T12:35:11.519 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (43099 ms) 2024-05-15T12:35:11.519 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: [----------] 1 test from neocls_handler_error (43099 ms total) 2024-05-15T12:35:11.519 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: 2024-05-15T12:35:11.519 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: [----------] Global test environment tear-down 2024-05-15T12:35:11.519 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: [==========] 1 test from 1 test suite ran. (43099 ms total) 2024-05-15T12:35:11.519 INFO:tasks.workunit.client.0.smithi049.stdout: handler_error: [ PASSED ] 1 test. 2024-05-15T12:35:11.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (19494 ms) 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (846 ms) 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (225 ms) 2024-05-15T12:35:11.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (1048 ms) 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (201 ms) 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (2818 ms) 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (81 ms) 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (2241 ms) 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (156 ms) 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (259 ms) 2024-05-15T12:35:11.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (126 ms) 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (164 ms) 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (421 ms) 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (427 ms) 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (470 ms) 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (100 ms) 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (29078 ms total) 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-05-15T12:35:11.590 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (43268 ms total) 2024-05-15T12:35:11.591 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-05-15T12:35:12.206 INFO:tasks.workunit.client.0.smithi049.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-05-15T12:35:12.206 INFO:tasks.workunit.client.0.smithi049.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-05-15T12:35:12.206 INFO:tasks.workunit.client.0.smithi049.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-05-15T12:35:12.206 INFO:tasks.workunit.client.0.smithi049.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-05-15T12:35:12.207 INFO:tasks.workunit.client.0.smithi049.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (43903 ms) 2024-05-15T12:35:12.207 INFO:tasks.workunit.client.0.smithi049.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (43903 ms total) 2024-05-15T12:35:12.207 INFO:tasks.workunit.client.0.smithi049.stdout: api_cls_remote_reads: 2024-05-15T12:35:12.207 INFO:tasks.workunit.client.0.smithi049.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-05-15T12:35:12.207 INFO:tasks.workunit.client.0.smithi049.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (43903 ms total) 2024-05-15T12:35:12.207 INFO:tasks.workunit.client.0.smithi049.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-05-15T12:35:14.074 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:03:43.284 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:14.152 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:03:43.320 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:14.157 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:03:43.367 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:14.158 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:03:43.368 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:14.177 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:03:43.345 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:14.267 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:03:43.476 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:14.420 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:03:43.588 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:14.516 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:03:43.726 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:14.840 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:03:44.050 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:20.698 INFO:tasks.workunit.client.0.smithi049.stdout: pool: Running main() from gmock_main.cc 2024-05-15T12:35:20.698 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-05-15T12:35:20.698 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [----------] Global test environment set-up. 2024-05-15T12:35:20.698 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ OK ] NeoRadosPools.PoolList (5122 ms) 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4247 ms) 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (6062 ms) 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (31692 ms) 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2488 ms) 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2582 ms) 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [----------] 6 tests from NeoRadosPools (52193 ms total) 2024-05-15T12:35:20.699 INFO:tasks.workunit.client.0.smithi049.stdout: pool: 2024-05-15T12:35:20.700 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [----------] Global test environment tear-down 2024-05-15T12:35:20.700 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [==========] 6 tests from 1 test suite ran. (52193 ms total) 2024-05-15T12:35:20.700 INFO:tasks.workunit.client.0.smithi049.stdout: pool: [ PASSED ] 6 tests. 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [----------] Global test environment set-up. 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: 2024-05-15T12:34:28.327+0000 7f2be6d1c780 -1 failed for service _ceph-mon._tcp 2024-05-15T12:35:27.288 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: 2024-05-15T12:34:28.327+0000 7f2be6d1c780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (270 ms) 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2028 ms) 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2298 ms total) 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb68cb7b0 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb68c7600 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6ba7db0 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4aa50 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4b4a0 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4bef0 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4c940 2024-05-15T12:35:27.289 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4d3f0 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4df30 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4ea70 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4f5b0 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c500f0 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c50c30 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c51770 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c522b0 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c52df0 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb757ee70 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4db80 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4dcf0 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb757eaa0 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb757ec50 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c52a00 2024-05-15T12:35:27.290 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c52c50 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c52030 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c51420 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4e680 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4e8e0 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c50960 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7582940 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4f250 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75834c0 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7583ee0 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4fe60 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7584a00 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75854e0 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7583c80 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb757f020 2024-05-15T12:35:27.291 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb757ed60 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c52140 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c51530 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c51640 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7584bb0 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c4f380 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7584cc0 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7584df0 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75833b0 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7584f40 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7582f00 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75850f0 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7585240 2024-05-15T12:35:27.292 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7585aa0 2024-05-15T12:35:40.104 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:04:09.313 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:40.185 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:04:09.395 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:40.190 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:04:09.358 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:40.195 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:04:09.364 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:40.273 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:04:09.483 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:40.286 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:04:09.496 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:40.404 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:04:09.572 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:40.488 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:04:09.697 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:40.522 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:04:09.731 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:35:42.640 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: starte watch_notify: Running main() from gmock_main.cc 2024-05-15T12:35:42.640 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-05-15T12:35:42.640 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [----------] Global test environment set-up. 2024-05-15T12:35:42.640 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-05-15T12:35:42.640 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-05-15T12:35:42.640 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: handle_notify cookie 94777341927584 notify_id 167503724544 notifier_gid 4528 2024-05-15T12:35:42.640 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (42806 ms) 2024-05-15T12:35:42.640 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: Trying... 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: handle_notify cookie 94777354060800 notify_id 244813135873 notifier_gid 4765 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: Waiting for 3.000000000s 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: Timed out. 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: Flushing... 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: Flushed... 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (31304 ms) 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (74110 ms total) 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [----------] Global test environment tear-down 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (74110 ms total) 2024-05-15T12:35:42.641 INFO:tasks.workunit.client.0.smithi049.stdout: watch_notify: [ PASSED ] 2 tests. 2024-05-15T12:35:45.283 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:45.271+0000 ea5d640 -1 osd.3 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4726.0:2 59.0 59:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-05-15T12:35:46.446 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:46.326+0000 ea5d640 -1 osd.3 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4726.0:2 59.0 59:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-05-15T12:35:47.574 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:47.579+0000 ea5d640 -1 osd.3 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4726.0:2 59.0 59:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-05-15T12:35:47.814 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:35:47.705+0000 12a65640 -1 osd.1 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4607.0:2 66.13 66:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-05-15T12:35:48.736 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:48.630+0000 ea5d640 -1 osd.3 66 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4726.0:2 59.0 59:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-05-15T12:35:49.980 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:49.846+0000 ea5d640 -1 osd.3 67 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4726.0:2 59.0 59:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-05-15T12:35:50.981 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:50.978+0000 ea5d640 -1 osd.3 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 62.0 62:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-05-15T12:35:52.335 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:52.212+0000 ea5d640 -1 osd.3 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 62.0 62:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-05-15T12:35:53.764 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:53.729+0000 ea5d640 -1 osd.3 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 62.0 62:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-05-15T12:35:54.867 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:54.799+0000 ea5d640 -1 osd.3 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 62.0 62:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-05-15T12:35:55.978 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: Running main() from gmock_main.cc 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [----------] Global test environment set-up. 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ OK ] LibRadosPools.PoolList (7249 ms) 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (5401 ms) 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (25244 ms) 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (11051 ms) 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4935 ms) 2024-05-15T12:35:55.979 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6202 ms) 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (8318 ms) 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (8525 ms) 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (10972 ms) 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [----------] 9 tests from LibRadosPools (87897 ms total) 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [----------] Global test environment tear-down 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (87897 ms total) 2024-05-15T12:35:55.980 INFO:tasks.workunit.client.0.smithi049.stdout: api_pool: [ PASSED ] 9 tests. 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify_test_cb 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (20603 ms) 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94611971339072 err -107 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (2414 ms) 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-05-15T12:35:55.985 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94611972071632 err -107 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1604 ms) 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_cb from 4407 notify_id 167503724544 cookie 94611972085808 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (217 ms) 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_cb from 4407 notify_id 167503724544 cookie 94611972085808 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (246 ms) 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_cb from 4407 notify_id 167503724545 cookie 94611972126864 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (2934 ms) 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_cb from 4407 notify_id 171798691840 cookie 94611972104752 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_cb from 4407 notify_id 171798691840 cookie 94611972138000 2024-05-15T12:35:55.986 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (1054 ms) 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_cb from 4407 notify_id 171798691841 cookie 94611972104752 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_cb from 4407 notify_id 171798691842 cookie 94611972104752 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3262 ms) 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94611972104752 err -107 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_cb from 4407 notify_id 176093659138 cookie 94611972104752 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (6485 ms) 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94611972104752 err -107 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (4669 ms) 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (43488 ms total) 2024-05-15T12:35:55.987 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: 2024-05-15T12:35:55.988 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-05-15T12:35:55.988 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-05-15T12:35:55.988 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: watch_notify_test_cb 2024-05-15T12:35:55.988 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (31037 ms) 2024-05-15T12:35:55.988 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (31037 ms total) 2024-05-15T12:35:55.988 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: 2024-05-15T12:35:55.988 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-05-15T12:35:55.988 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (87809 ms total) 2024-05-15T12:35:55.988 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-05-15T12:35:56.179 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:56.184+0000 ea5d640 -1 osd.3 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 62.0 62:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-05-15T12:35:57.370 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:57.280+0000 ea5d640 -1 osd.3 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 62.0 62:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-05-15T12:35:58.391 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:58.313+0000 ea5d640 -1 osd.3 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 62.0 62:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-05-15T12:35:59.682 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:35:59.663+0000 ea5d640 -1 osd.3 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 62.0 62:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-05-15T12:36:07.185 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:36:07.147+0000 ea5d640 -1 osd.2 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4580.0:5 60.8 60.3f1ee208 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:36:08.266 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:36:08.209+0000 ea5d640 -1 osd.2 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4580.0:5 60.8 60.3f1ee208 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:36:08.295 INFO:tasks.workunit.client.0.smithi049.stdout:d 0x55fdb7586100 2024-05-15T12:36:08.295 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7586740 2024-05-15T12:36:08.295 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7586d20 2024-05-15T12:36:08.295 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7586ed0 2024-05-15T12:36:08.295 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7586fe0 2024-05-15T12:36:08.295 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75870f0 2024-05-15T12:36:08.295 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7583ff0 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7584610 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7584780 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7586290 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7585370 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75863a0 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75864b0 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7586600 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75835d0 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7583b40 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75836e0 2024-05-15T12:36:08.296 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7583830 2024-05-15T12:36:08.297 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75803f0 2024-05-15T12:36:08.297 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75855f0 2024-05-15T12:36:08.297 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7583010 2024-05-15T12:36:08.297 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7583120 2024-05-15T12:36:08.297 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7583230 2024-05-15T12:36:08.297 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7583960 2024-05-15T12:36:08.297 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7583d90 2024-05-15T12:36:08.298 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75848b0 2024-05-15T12:36:08.298 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7582a50 2024-05-15T12:36:08.298 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7585700 2024-05-15T12:36:08.298 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7585810 2024-05-15T12:36:08.298 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7585920 2024-05-15T12:36:08.299 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7584140 2024-05-15T12:36:08.299 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7582b60 2024-05-15T12:36:08.299 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7582c70 2024-05-15T12:36:08.299 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7582da0 2024-05-15T12:36:08.299 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c41600 2024-05-15T12:36:08.299 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb757f5e0 2024-05-15T12:36:08.300 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb757ff80 2024-05-15T12:36:08.300 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb75800d0 2024-05-15T12:36:08.300 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb7580200 2024-05-15T12:36:08.300 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c41710 2024-05-15T12:36:08.300 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c41880 2024-05-15T12:36:08.300 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 0x55fdb6c41a30 2024-05-15T12:36:08.301 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: started 93 aios 2024-05-15T12:36:08.301 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb68cb7b0 2024-05-15T12:36:08.301 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb68c7600 2024-05-15T12:36:08.301 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6ba7db0 2024-05-15T12:36:08.301 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4aa50 2024-05-15T12:36:08.302 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4b4a0 2024-05-15T12:36:08.302 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4bef0 2024-05-15T12:36:08.302 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4c940 2024-05-15T12:36:08.302 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4d3f0 2024-05-15T12:36:08.302 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4df30 2024-05-15T12:36:08.302 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4ea70 2024-05-15T12:36:08.303 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4f5b0 2024-05-15T12:36:08.303 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c500f0 2024-05-15T12:36:08.303 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c50c30 2024-05-15T12:36:08.303 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c51770 2024-05-15T12:36:08.303 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c522b0 2024-05-15T12:36:08.303 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c52df0 2024-05-15T12:36:08.304 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb757ee70 2024-05-15T12:36:08.304 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4db80 2024-05-15T12:36:08.304 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4dcf0 2024-05-15T12:36:08.304 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb757eaa0 2024-05-15T12:36:08.304 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb757ec50 2024-05-15T12:36:08.304 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c52a00 2024-05-15T12:36:08.304 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c52c50 2024-05-15T12:36:08.305 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c52030 2024-05-15T12:36:08.305 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c51420 2024-05-15T12:36:08.305 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4e680 2024-05-15T12:36:08.305 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4e8e0 2024-05-15T12:36:08.305 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c50960 2024-05-15T12:36:08.305 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7582940 2024-05-15T12:36:08.306 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4f250 2024-05-15T12:36:08.306 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75834c0 2024-05-15T12:36:08.306 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7583ee0 2024-05-15T12:36:08.306 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4fe60 2024-05-15T12:36:08.306 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7584a00 2024-05-15T12:36:08.306 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75854e0 2024-05-15T12:36:08.307 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7583c80 2024-05-15T12:36:08.307 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb757f020 2024-05-15T12:36:08.307 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb757ed60 2024-05-15T12:36:08.307 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c52140 2024-05-15T12:36:09.665 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:36:09.668+0000 ea5d640 -1 osd.2 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4580.0:5 60.8 60.3f1ee208 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:36:11.325 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:36:11.333+0000 ea5d640 -1 osd.2 86 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4580.0:5 60.8 60.3f1ee208 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:36:12.633 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:36:12.633+0000 ea5d640 -1 osd.2 87 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4580.0:5 60.8 60.3f1ee208 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:36:13.840 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:36:13.772+0000 ea5d640 -1 osd.2 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4580.0:5 60.8 60.3f1ee208 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53) 2024-05-15T12:36:18.538 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:18.435+0000 12a65640 -1 osd.1 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:20.127 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:04:49.336 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:20.143 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:04:49.353 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:20.202 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:04:49.370 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:20.212 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:04:49.422 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:20.218 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:04:49.386 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:20.232 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:04:49.442 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:20.403 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:20.386+0000 12a65640 -1 osd.1 91 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:20.418 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:04:49.586 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:20.479 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:04:49.689 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:20.678 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:04:49.887 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:21.598 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:21.449+0000 12a65640 -1 osd.1 93 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:22.648 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:22.621+0000 12a65640 -1 osd.1 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:23.103 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting cls: Running main() from gmock_main.cc 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [----------] Global test environment set-up. 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [ OK ] NeoRadosCls.DNE (42933 ms) 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (71775 ms) 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [----------] 2 tests from NeoRadosCls (114708 ms total) 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [----------] Global test environment tear-down 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [==========] 2 tests from 1 test suite ran. (114708 ms total) 2024-05-15T12:36:23.104 INFO:tasks.workunit.client.0.smithi049.stdout: cls: [ PASSED ] 2 tests. 2024-05-15T12:36:23.925 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:23.871+0000 12a65640 -1 osd.1 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:24.908 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:24.897+0000 12a65640 -1 osd.1 96 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:26.197 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:26.036+0000 12a65640 -1 osd.1 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:27.082 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:27.080+0000 12a65640 -1 osd.1 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:28.365 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:28.235+0000 12a65640 -1 osd.1 99 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:30.695 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:30.669+0000 12a65640 -1 osd.1 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:32.152 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:32.053+0000 12a65640 -1 osd.1 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74.1272949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:33.565 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:33.497+0000 12a65640 -1 osd.1 101 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:34.878 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T12:36:34.828+0000 12a65640 -1 osd.1 102 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4789.0:2 74.1s0 74:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64) 2024-05-15T12:36:40.450 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:05:09.659 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:40.455 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:05:09.665 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:40.522 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:05:09.690 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:40.532 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:05:09.700 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:40.541 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:05:09.750 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:40.542 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:05:09.752 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:40.648 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:05:09.816 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:40.652 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:05:09.861 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:40.729 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:05:09.938 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:36:46.790 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:36:46.695+0000 ea5d640 -1 osd.3 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:6 64.d 64:b3c717c4:::4.obj:head [write 0~183 in=183b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) 2024-05-15T12:36:47.798 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:36:47.779+0000 ea5d640 -1 osd.3 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:6 64.d 64:b3c717c4:::4.obj:head [write 0~183 in=183b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) 2024-05-15T12:36:49.085 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-15T12:36:49.073+0000 ea5d640 -1 osd.4 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4634.0:2 77.1s0 77:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) 2024-05-15T12:36:49.129 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:36:48.982+0000 ea5d640 -1 osd.0 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:36:49.244 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T12:36:49.239+0000 ea5d640 -1 osd.3 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:6 64.d 64:b3c717c4:::4.obj:head [write 0~183 in=183b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) 2024-05-15T12:36:50.491 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:36:50.500+0000 ea5d640 -1 osd.0 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:36:51.368 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-15T12:36:51.277+0000 ea5d640 -1 osd.4 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4634.0:2 77.1s0 77:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) 2024-05-15T12:36:51.894 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:36:51.875+0000 ea5d640 -1 osd.0 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:36:52.515 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-15T12:36:52.393+0000 ea5d640 -1 osd.4 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4634.0:2 77.1s0 77:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) 2024-05-15T12:36:52.885 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:36:52.892+0000 ea5d640 -1 osd.0 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:36:54.088 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:36:54.091+0000 ea5d640 -1 osd.0 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:36:55.347 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:36:55.326+0000 ea5d640 -1 osd.0 123 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:36:56.442 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:36:56.356+0000 ea5d640 -1 osd.0 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:36:57.540 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:36:57.546+0000 ea5d640 -1 osd.0 125 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:36:58.599 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:36:58.604+0000 ea5d640 -1 osd.0 127 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:36:59.595 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:36:59.597+0000 ea5d640 -1 osd.0 127 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:00.709 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:05:29.918 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:00.715 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:05:29.925 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:00.788 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:05:29.956 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:00.792 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:05:29.960 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:00.801 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:05:30.011 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:00.816 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:05:30.025 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:00.910 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:05:30.119 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:00.921 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:05:30.089 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:00.925 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:05:30.135 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:02.105 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:02.073+0000 ea5d640 -1 osd.0 130 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:03.139 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:03.145+0000 ea5d640 -1 osd.0 131 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:04.124 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:04.126+0000 ea5d640 -1 osd.0 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:05.437 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:05.433+0000 ea5d640 -1 osd.0 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: seed 40411 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (20684 ms) 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (3259 ms) 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-05-15T12:37:06.066 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (260 ms) 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (658 ms) 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (3812 ms) 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (28673 ms total) 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (80062 ms) 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (39 ms) 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1775 ms) 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-05-15T12:37:06.067 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (21959 ms) 2024-05-15T12:37:06.068 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (103835 ms total) 2024-05-15T12:37:06.068 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: 2024-05-15T12:37:06.068 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-05-15T12:37:06.068 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (157905 ms total) 2024-05-15T12:37:06.068 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-05-15T12:37:06.721 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:06.726+0000 ea5d640 -1 osd.0 134 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:07.843 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:07.774+0000 ea5d640 -1 osd.0 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:09.414 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:09.417+0000 ea5d640 -1 osd.0 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:09.697 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-05-15T12:37:09.697 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-05-15T12:37:09.697 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-05-15T12:37:09.697 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (800 ms) 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (599 ms) 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (160093 ms) 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (161492 ms total) 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (161492 ms total) 2024-05-15T12:37:09.698 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-05-15T12:37:10.785 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:10.746+0000 ea5d640 -1 osd.0 137 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:11.936 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:11.901+0000 ea5d640 -1 osd.0 138 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:13.184 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:13.101+0000 ea5d640 -1 osd.0 139 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:14.255 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:14.237+0000 ea5d640 -1 osd.0 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:15.430 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:15.278+0000 ea5d640 -1 osd.0 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96.f792e973 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:16.662 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:16.526+0000 ea5d640 -1 osd.0 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:17.581 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:17.587+0000 ea5d640 -1 osd.0 143 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:18.659 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:18.665+0000 ea5d640 -1 osd.0 143 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:19.887 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:19.795+0000 ea5d640 -1 osd.0 145 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:21.110 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:05:50.320 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:21.112 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:05:50.322 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:21.116 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:21.106+0000 ea5d640 -1 osd.0 146 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:21.193 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:05:50.360 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:21.194 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:05:50.362 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:21.242 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:05:50.452 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:21.246 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:05:50.455 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:21.300 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:05:50.468 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:21.305 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:05:50.515 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:21.363 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:05:50.572 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:22.278 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:22.122+0000 ea5d640 -1 osd.0 147 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:22.584 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: Running main() from gmock_main.cc 2024-05-15T12:37:22.584 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-05-15T12:37:22.584 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [----------] Global test environment set-up. 2024-05-15T12:37:22.584 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-05-15T12:37:22.584 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-05-15T12:37:22.584 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ OK ] LibRadosStat.Stat (22832 ms) 2024-05-15T12:37:22.584 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-05-15T12:37:22.584 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (1192 ms) 2024-05-15T12:37:22.584 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ OK ] LibRadosStat.StatNS (375 ms) 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (3 ms) 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (206 ms) 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [----------] 5 tests from LibRadosStat (24608 ms total) 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (56568 ms) 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (51085 ms) 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-05-15T12:37:22.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (17 ms) 2024-05-15T12:37:22.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-05-15T12:37:22.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (13499 ms) 2024-05-15T12:37:22.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (121169 ms total) 2024-05-15T12:37:22.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: 2024-05-15T12:37:22.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [----------] Global test environment tear-down 2024-05-15T12:37:22.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (174441 ms total) 2024-05-15T12:37:22.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_stat: [ PASSED ] 9 tests. 2024-05-15T12:37:23.653 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:23.566+0000 ea5d640 -1 osd.0 148 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:24.614 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:24.555+0000 ea5d640 -1 osd.0 150 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:25.688 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:25.637+0000 ea5d640 -1 osd.0 151 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:26.708 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:26.694+0000 ea5d640 -1 osd.0 152 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:27.728 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:27.733+0000 ea5d640 -1 osd.0 153 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:31.520 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:31.423+0000 ea5d640 -1 osd.0 156 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:32.584 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:32.584+0000 ea5d640 -1 osd.0 157 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:33.615 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:37:33.619+0000 ea5d640 -1 osd.0 158 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4694.0:2 96.13 96:ce9749ef:test-rados-api-smithi049-40136-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-05-15T12:37:34.878 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: Running main() from gmock_main.cc 2024-05-15T12:37:34.878 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [----------] Global test environment set-up. 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1159 ms) 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (137 ms) 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (181343 ms) 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:21.108633+0000 osd.0 [WRN] 2 slow requests (by type [ 'waiting for sub ops' : 2 ] most affected pool [ 'LibRadosMisc_smithi049-40136-1' : 1 ]) 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:22.123820+0000 osd.0 [WRN] 2 slow requests (by type [ 'waiting for sub ops' : 2 ] most affected pool [ 'LibRadosMisc_smithi049-40136-1' : 1 ]) 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:29.402154+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 24796.413ms) (OSD_SLOW_PING_TIME_BACK) 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:29.402629+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 21427.064ms) (OSD_SLOW_PING_TIME_FRONT) 2024-05-15T12:37:34.879 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:29.584109+0000 mon.b [INF] from='client.? 172.21.15.49:0/787268343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi049-40625-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:37:34.880 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:29.603879+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi049-40625-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:37:34.881 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:23.568063+0000 osd.0 [WRN] 2 slow requests (by type [ 'waiting for sub ops' : 2 ] most affected pool [ 'LibRadosMisc_smithi049-40136-1' : 1 ]) 2024-05-15T12:37:34.881 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:24.557659+0000 osd.0 [WRN] 2 slow requests (by type [ 'waiting for sub ops' : 2 ] most affected pool [ 'LibRadosMisc_smithi049-40136-1' : 1 ]) 2024-05-15T12:37:34.881 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:25.639693+0000 osd.0 [WRN] 2 slow requests (by type [ 'waiting for sub ops' : 2 ] most affected pool [ 'LibRadosMisc_smithi049-40136-1' : 1 ]) 2024-05-15T12:37:34.881 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:30.629003+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi049-40625-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-05-15T12:37:34.881 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:30.917186+0000 mon.b [INF] from='client.? 172.21.15.49:0/341948564' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-05-15T12:37:34.881 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:30.952104+0000 client.admin [INF] onexx 2024-05-15T12:37:34.881 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:30.959167+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-05-15T12:37:34.881 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:30.994991+0000 mon.a [WRN] Health check update: 2 slow ops, oldest one blocked for 68 sec, osd.0 has slow ops (SLOW_OPS) 2024-05-15T12:37:34.881 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:31.423818+0000 osd.0 [WRN] 1 slow requests (by type [ 'waiting for sub ops' : 1 ] most affected pool [ 'LibRadosMisc_smithi049-40136-1' : 1 ]) 2024-05-15T12:37:34.881 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:32.426528+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-05-15T12:37:34.882 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:34.498089+0000 mon.b [INF] from='client.? 172.21.15.49:0/341948564' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-05-15T12:37:37.891 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:34.564683+0000 client.admin [INF] threexx 2024-05-15T12:37:37.892 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: got: 2024-05-15T12:37:34.581830+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-05-15T12:37:37.892 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7062 ms) 2024-05-15T12:37:37.892 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (189701 ms total) 2024-05-15T12:37:37.892 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: 2024-05-15T12:37:37.892 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [----------] Global test environment tear-down 2024-05-15T12:37:37.892 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (189701 ms total) 2024-05-15T12:37:37.892 INFO:tasks.workunit.client.0.smithi049.stdout: api_cmd: [ PASSED ] 4 tests. 2024-05-15T12:37:41.200 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:06:10.409 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:41.271 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:06:10.480 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:41.282 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:06:10.450 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:41.285 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:06:10.453 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:41.343 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:06:10.552 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:41.345 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:06:10.554 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:41.366 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:06:10.575 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:41.370 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:06:10.538 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:41.436 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:06:10.645 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: Running main() from gmock_main.cc 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [----------] Global test environment set-up. 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: seed 40030 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (27837 ms) 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-05-15T12:37:47.196 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (498 ms) 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (1049 ms) 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (597 ms) 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (44 ms) 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (55 ms) 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (79 ms) 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (414 ms) 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (86 ms) 2024-05-15T12:37:47.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (218 ms) 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (92 ms) 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (221 ms) 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (116 ms) 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (212 ms) 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (446 ms) 2024-05-15T12:37:47.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (260 ms) 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (68 ms) 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (78 ms) 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (153 ms) 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (32523 ms total) 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (84802 ms) 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-05-15T12:37:47.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (20741 ms) 2024-05-15T12:37:47.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-05-15T12:37:47.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (7820 ms) 2024-05-15T12:37:47.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-05-15T12:37:47.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (17673 ms) 2024-05-15T12:37:47.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-05-15T12:37:47.301 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:37:47.301+0000 12a65640 -1 osd.5 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4634.0:5 77.6s0 77:6ce5201f:test-rados-api-smithi049-40013-18::bar:head [write 19~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140) 2024-05-15T12:37:48.406 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:37:48.407+0000 12a65640 -1 osd.5 173 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4634.0:5 77.6s0 77:6ce5201f:test-rados-api-smithi049-40013-18::bar:head [write 19~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140) 2024-05-15T12:37:57.779 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (16 api_lock_pp: Running main() from gmock_main.cc 2024-05-15T12:37:57.779 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-05-15T12:37:57.779 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-05-15T12:37:57.779 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-05-15T12:37:57.779 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: seed 40111 2024-05-15T12:37:57.779 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-05-15T12:37:57.779 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (17038 ms) 2024-05-15T12:37:57.779 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-05-15T12:37:57.779 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (6750 ms) 2024-05-15T12:37:57.779 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1390 ms) 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (4206 ms) 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (1545 ms) 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (199 ms) 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (214 ms) 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (131 ms) 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (31473 ms total) 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-05-15T12:37:57.780 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (27700 ms) 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (1768 ms) 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (86524 ms) 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (17847 ms) 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (9994 ms) 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (2710 ms) 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (2568 ms) 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (5644 ms) 2024-05-15T12:37:57.781 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (154755 ms total) 2024-05-15T12:37:57.782 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: 2024-05-15T12:37:57.782 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-05-15T12:37:57.782 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (209756 ms total) 2024-05-15T12:37:57.782 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-05-15T12:38:01.520 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:06:30.729 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:01.523 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:06:30.732 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:01.607 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:06:30.775 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:01.610 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:06:30.778 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:01.611 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:06:30.820 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:01.620 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:06:30.829 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:01.634 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:06:30.843 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:01.703 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:06:30.870 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:01.729 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:06:30.938 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:12.669 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: notify 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (20444 ms) 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (14367 ms) 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (34811 ms total) 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-05-15T12:38:12.670 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: notify 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (111593 ms) 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: notify 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (10180 ms) 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (21513 ms) 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (3353 ms) 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: handle_notify cookie 94268380959392 notify_id 751619276803 notifier_gid 4846 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (4106 ms) 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-05-15T12:38:12.671 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: handle_notify cookie 94268380959392 notify_id 816043786242 notifier_gid 4846 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (16772 ms) 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: handle_notify cookie 94268382500160 notify_id 816043786243 notifier_gid 4846 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (750 ms) 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: handle_notify cookie 94268382500160 notify_id 816043786244 notifier_gid 4846 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (1032 ms) 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: handle_notify cookie 94268382500160 notify_id 816043786244 notifier_gid 4846 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (429 ms) 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: handle_notify cookie 94268382500160 notify_id 816043786245 notifier_gid 4846 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (655 ms) 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-05-15T12:38:12.672 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: trying... 2024-05-15T12:38:12.673 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: handle_notify cookie 94268382500160 notify_id 816043786243 notifier_gid 4846 2024-05-15T12:38:12.673 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: timed out 2024-05-15T12:38:18.313 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp api_list: [==========] Running 11 tests from 3 test suites. 2024-05-15T12:38:18.313 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [----------] Global test environment set-up. 2024-05-15T12:38:18.314 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [----------] 7 tests from LibRadosList 2024-05-15T12:38:18.314 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-05-15T12:38:18.314 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosList.ListObjects (27130 ms) 2024-05-15T12:38:18.314 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-05-15T12:38:18.314 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (1041 ms) 2024-05-15T12:38:18.314 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-05-15T12:38:18.314 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: myset foo1,foo2,foo3 2024-05-15T12:38:18.315 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo1 2024-05-15T12:38:18.315 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo2 2024-05-15T12:38:18.315 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo3 2024-05-15T12:38:18.315 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: myset foo1,foo4,foo5 2024-05-15T12:38:18.315 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo4 2024-05-15T12:38:18.315 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo5 2024-05-15T12:38:18.315 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo1 2024-05-15T12:38:18.315 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: myset foo6,foo7 2024-05-15T12:38:18.315 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo7 2024-05-15T12:38:18.316 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo6 2024-05-15T12:38:18.316 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-05-15T12:38:18.316 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: ns1:foo4 2024-05-15T12:38:18.316 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: ns1:foo5 2024-05-15T12:38:18.316 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: ns2:foo7 2024-05-15T12:38:18.316 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: ns2:foo6 2024-05-15T12:38:18.316 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: ns1:foo1 2024-05-15T12:38:18.316 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: :foo1 2024-05-15T12:38:18.316 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: :foo2 2024-05-15T12:38:18.317 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: :foo3 2024-05-15T12:38:18.317 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (136079 ms) 2024-05-15T12:38:18.317 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-05-15T12:38:18.317 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 1 0 2024-05-15T12:38:18.317 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 10 0 2024-05-15T12:38:18.317 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 13 0 2024-05-15T12:38:18.317 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 7 0 2024-05-15T12:38:18.317 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 14 0 2024-05-15T12:38:18.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 0 0 2024-05-15T12:38:18.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 15 0 2024-05-15T12:38:18.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 11 0 2024-05-15T12:38:18.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 5 0 2024-05-15T12:38:18.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 8 0 2024-05-15T12:38:18.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 6 0 2024-05-15T12:38:18.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 3 0 2024-05-15T12:38:18.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 4 0 2024-05-15T12:38:18.318 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 12 0 2024-05-15T12:38:18.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 9 0 2024-05-15T12:38:18.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 2 0 2024-05-15T12:38:18.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-05-15T12:38:18.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (52874 ms) 2024-05-15T12:38:18.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-05-15T12:38:18.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: x cursor=MIN 2024-05-15T12:38:18.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=1 cursor=19:02547ec2:::1:head 2024-05-15T12:38:18.319 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=10 cursor=19:52ea6a34:::10:head 2024-05-15T12:38:18.320 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=13 cursor=19:566253c9:::13:head 2024-05-15T12:38:18.320 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=7 cursor=19:5c6b0b28:::7:head 2024-05-15T12:38:18.320 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=14 cursor=19:62a1935d:::14:head 2024-05-15T12:38:18.320 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=0 cursor=19:6cac518f:::0:head 2024-05-15T12:38:18.320 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=15 cursor=19:863748b0:::15:head 2024-05-15T12:38:18.320 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=11 cursor=19:89d3ae78:::11:head 2024-05-15T12:38:18.320 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=5 cursor=19:b29083e3:::5:head 2024-05-15T12:38:18.320 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=8 cursor=19:bd63b0f1:::8:head 2024-05-15T12:38:18.320 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=6 cursor=19:c4fdafeb:::6:head 2024-05-15T12:38:18.321 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=3 cursor=19:cfc208b3:::3:head 2024-05-15T12:38:18.321 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=4 cursor=19:d83876eb:::4:head 2024-05-15T12:38:18.321 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=12 cursor=19:de5d7c5f:::12:head 2024-05-15T12:38:18.321 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=9 cursor=19:e960b815:::9:head 2024-05-15T12:38:18.321 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > oid=2 cursor=19:f905c69b:::2:head 2024-05-15T12:38:18.321 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: FIRST> seek to MIN oid=1 2024-05-15T12:38:18.321 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=1 cursor=19:02547ec2:::1:head 2024-05-15T12:38:18.321 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:02547ec2:::1:head 2024-05-15T12:38:18.321 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:02547ec2:::1:head -> 1 2024-05-15T12:38:18.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=10 cursor=19:52ea6a34:::10:head 2024-05-15T12:38:18.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:52ea6a34:::10:head 2024-05-15T12:38:18.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:52ea6a34:::10:head -> 10 2024-05-15T12:38:18.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=13 cursor=19:566253c9:::13:head 2024-05-15T12:38:18.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:566253c9:::13:head 2024-05-15T12:38:18.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:566253c9:::13:head -> 13 2024-05-15T12:38:18.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=7 cursor=19:5c6b0b28:::7:head 2024-05-15T12:38:18.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:5c6b0b28:::7:head 2024-05-15T12:38:18.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:5c6b0b28:::7:head -> 7 2024-05-15T12:38:18.606 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : open_pools_parallel: process_1_[40933]: starting. 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_1_[40933]: creating pool ceph_test_rados_open_pools_parallel.smithi049-40885 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_1_[40933]: created object 0... 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_1_[40933]: created object 25... 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_1_[40933]: created object 49... 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_1_[40933]: finishing. 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_1_[40933]: shutting down. 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_2_[40935]: starting. 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_2_[40935]: rados_pool_create. 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_2_[40935]: rados_ioctx_create. 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_2_[40935]: shutting down. 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: ******************************* 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_3_[42313]: starting. 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_3_[42313]: creating pool ceph_test_rados_open_pools_parallel.smithi049-40885 2024-05-15T12:38:18.607 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_3_[42313]: created object 0... 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_3_[42313]: created object 25... 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_3_[42313]: created object 49... 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_3_[42313]: finishing. 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_3_[42313]: shutting down. 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: ******************************* 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_4_[42314]: starting. 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_4_[42314]: rados_pool_create. 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_4_[42314]: rados_ioctx_create. 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: process_4_[42314]: shutting down. 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: ******************************* 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: ******************************* 2024-05-15T12:38:18.608 INFO:tasks.workunit.client.0.smithi049.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-05-15T12:38:25.146 INFO:tasks.workunit.client.0.smithi049.stdout: oid=14 cursor=19:62a1935d:::14:head 2024-05-15T12:38:25.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:62a1935d:::14:head 2024-05-15T12:38:25.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:62a1935d:::14:head -> 14 2024-05-15T12:38:25.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=0 cursor=19:6cac518f:::0:head 2024-05-15T12:38:25.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:6cac518f:::0:head 2024-05-15T12:38:25.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:6cac518f:::0:head -> 0 2024-05-15T12:38:25.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=15 cursor=19:863748b0:::15:head 2024-05-15T12:38:25.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:863748b0:::15:head 2024-05-15T12:38:25.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:863748b0:::15:head -> 15 2024-05-15T12:38:25.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=11 cursor=19:89d3ae78:::11:head 2024-05-15T12:38:25.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:89d3ae78:::11:head 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:89d3ae78:::11:head -> 11 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=5 cursor=19:b29083e3:::5:head 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:b29083e3:::5:head 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:b29083e3:::5:head -> 5 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=8 cursor=19:bd63b0f1:::8:head 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:bd63b0f1:::8:head 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:bd63b0f1:::8:head -> 8 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=6 cursor=19:c4fdafeb:::6:head 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:c4fdafeb:::6:head 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:c4fdafeb:::6:head -> 6 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=3 cursor=19:cfc208b3:::3:head 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:cfc208b3:::3:head 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:cfc208b3:::3:head -> 3 2024-05-15T12:38:25.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=4 cursor=19:d83876eb:::4:head 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:d83876eb:::4:head 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:d83876eb:::4:head -> 4 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=12 cursor=19:de5d7c5f:::12:head 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:de5d7c5f:::12:head 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:de5d7c5f:::12:head -> 12 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=9 cursor=19:e960b815:::9:head 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:e960b815:::9:head 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:e960b815:::9:head -> 9 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : oid=2 cursor=19:f905c69b:::2:head 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:f905c69b:::2:head 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:f905c69b:::2:head -> 2 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:62a1935d:::14:head 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:62a1935d:::14:head expected=19:62a1935d:::14:head 2024-05-15T12:38:25.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:62a1935d:::14:head -> 14 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=14 expected=14 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:5c6b0b28:::7:head 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:5c6b0b28:::7:head expected=19:5c6b0b28:::7:head 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:5c6b0b28:::7:head -> 7 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=7 expected=7 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:566253c9:::13:head 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:566253c9:::13:head expected=19:566253c9:::13:head 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:566253c9:::13:head -> 13 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=13 expected=13 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:52ea6a34:::10:head 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:52ea6a34:::10:head expected=19:52ea6a34:::10:head 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:52ea6a34:::10:head -> 10 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=10 expected=10 2024-05-15T12:38:25.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:f905c69b:::2:head 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:f905c69b:::2:head expected=19:f905c69b:::2:head 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:f905c69b:::2:head -> 2 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=2 expected=2 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:cfc208b3:::3:head 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:cfc208b3:::3:head expected=19:cfc208b3:::3:head 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:cfc208b3:::3:head -> 3 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=3 expected=3 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:863748b0:::15:head 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:863748b0:::15:head expected=19:863748b0:::15:head 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:863748b0:::15:head -> 15 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=15 expected=15 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:e960b815:::9:head 2024-05-15T12:38:25.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:e960b815:::9:head expected=19:e960b815:::9:head 2024-05-15T12:38:25.152 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:e960b815:::9:head -> 9 2024-05-15T12:38:25.780 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: Running main() from gmock_main.cc 2024-05-15T12:38:25.780 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-05-15T12:38:25.780 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [----------] Global test environment set-up. 2024-05-15T12:38:25.780 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-05-15T12:38:25.780 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-05-15T12:38:25.780 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6760 ms) 2024-05-15T12:38:25.780 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-05-15T12:38:25.780 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (34659 ms) 2024-05-15T12:38:25.781 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-05-15T12:38:25.781 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (195953 ms) 2024-05-15T12:38:25.781 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [----------] 3 tests from NeoRadosCmd (237372 ms total) 2024-05-15T12:38:25.781 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: 2024-05-15T12:38:25.781 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [----------] Global test environment tear-down 2024-05-15T12:38:25.781 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [==========] 3 tests from 1 test suite ran. (237372 ms total) 2024-05-15T12:38:25.781 INFO:tasks.workunit.client.0.smithi049.stdout: cmd: [ PASSED ] 3 tests. 2024-05-15T12:38:28.548 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_1_[40789]: starting. 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_1_[40789]: creating pool ceph_test_rados_delete_pools_parallel.smithi049-40761 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_1_[40789]: created object 0... 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_1_[40789]: created object 25... 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_1_[40789]: created object 49... 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_1_[40789]: finishing. 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_1_[40789]: shutting down. 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_2_[40790]: starting. 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_2_[40790]: deleting pool ceph_test_rados_delete_pools_parallel.smithi049-40761 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_2_[40790]: shutting down. 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: ******************************* 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_3_[42440]: starting. 2024-05-15T12:38:28.549 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_3_[42440]: creating pool ceph_test_rados_delete_pools_parallel.smithi049-40761 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_3_[42440]: created object 0... 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_3_[42440]: created object 25... 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_3_[42440]: created object 49... 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_3_[42440]: finishing. 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_3_[42440]: shutting down. 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: ******************************* 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_5_[42442]: starting. 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_5_[42442]: listing objects. 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_5_[42442]: listed object 0... 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_5_[42442]: listed object 25... 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_5_[42442]: saw 50 objects 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_5_[42442]: shutting down. 2024-05-15T12:38:28.550 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: ******************************* 2024-05-15T12:38:28.551 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_4_[42441]: starting. 2024-05-15T12:38:28.551 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_4_[42441]: deleting pool ceph_test_rados_delete_pools_parallel.smithi049-40761 2024-05-15T12:38:28.551 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: process_4_[42441]: shutting down. 2024-05-15T12:38:28.551 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: ******************************* 2024-05-15T12:38:28.551 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: ******************************* 2024-05-15T12:38:28.551 INFO:tasks.workunit.client.0.smithi049.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-05-15T12:38:29.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: Running main() from gmock_main.cc 2024-05-15T12:38:29.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-05-15T12:38:29.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [----------] Global test environment set-up. 2024-05-15T12:38:29.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-05-15T12:38:29.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-05-15T12:38:29.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (19237 ms) 2024-05-15T12:38:29.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-05-15T12:38:29.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLock.LockShared (1621 ms) 2024-05-15T12:38:29.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-05-15T12:38:29.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (2133 ms) 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1169 ms) 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (2830 ms) 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLock.Unlock (647 ms) 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (1795 ms) 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (182 ms) 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [----------] 8 tests from LibRadosLock (29614 ms total) 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-05-15T12:38:29.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (88667 ms) 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (6347 ms) 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (5599 ms) 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (7003 ms) 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (59388 ms) 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (1877 ms) 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (1404 ms) 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-05-15T12:38:29.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (210 ms) 2024-05-15T12:38:29.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (170495 ms total) 2024-05-15T12:38:29.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: 2024-05-15T12:38:29.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [----------] Global test environment tear-down 2024-05-15T12:38:29.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (241129 ms total) 2024-05-15T12:38:29.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_lock: [ PASSED ] 16 tests. 2024-05-15T12:38:35.203 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_1_[40707]: starting. 2024-05-15T12:38:35.203 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_1_[40707]: creating pool ceph_test_rados_list_parallel.smithi049-40671 2024-05-15T12:38:35.203 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_1_[40707]: created object 0... 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_1_[40707]: created object 25... 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_1_[40707]: created object 49... 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_1_[40707]: finishing. 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_1_[40707]: shutting down. 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_2_[40710]: starting. 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_2_[40710]: listing objects. 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_2_[40710]: listed object 0... 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_2_[40710]: listed object 25... 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_2_[40710]: saw 50 objects 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_2_[40710]: shutting down. 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_3_[42210]: starting. 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_3_[42210]: creating pool ceph_test_rados_list_parallel.smithi049-40671 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_3_[42210]: created object 0... 2024-05-15T12:38:35.204 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_3_[42210]: created object 25... 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_3_[42210]: created object 49... 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_3_[42210]: finishing. 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_3_[42210]: shutting down. 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_4_[42211]: starting. 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_4_[42211]: listing objects. 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_4_[42211]: listed object 0... 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_4_[42211]: listed object 25... 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_4_[42211]: saw 44 objects 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_4_[42211]: shutting down. 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_5_[42212]: starting. 2024-05-15T12:38:35.205 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_5_[42212]: removed 25 objects... 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_5_[42212]: removed half of the objects 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_5_[42212]: removed 50 objects... 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_5_[42212]: removed 50 objects 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_5_[42212]: shutting down. 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_6_[51925]: starting. 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_6_[51925]: creating pool ceph_test_rados_list_parallel.smithi049-40671 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_6_[51925]: created object 0... 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_6_[51925]: created object 25... 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_6_[51925]: created object 49... 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_6_[51925]: finishing. 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_6_[51925]: shutting down. 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:35.206 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:35.207 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_7_[51926]: starting. 2024-05-15T12:38:35.207 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_7_[51926]: listing objects. 2024-05-15T12:38:35.207 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_7_[51926]: listed object 0... 2024-05-15T12:38:35.207 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_7_[51926]: listed object 25... 2024-05-15T12:38:35.207 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_7_[51926]: listed object 50... 2024-05-15T12:38:35.207 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_7_[51926]: saw 55 objects 2024-05-15T12:38:35.207 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_7_[51926]: shutting down. 2024-05-15T12:38:35.207 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:35.208 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:35.208 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_8_[51927]: starting. 2024-05-15T12:38:35.208 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_8_[51927]: added 25 objects... 2024-05-15T12:38:35.208 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_8_[51927]: added half of the objects 2024-05-15T12:38:35.208 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_8_[51927]: added 50 objects... 2024-05-15T12:38:35.208 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_8_[51927]: added 50 objects 2024-05-15T12:38:35.208 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_8_[51927]: shutting down. 2024-05-15T12:38:35.208 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:35.208 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:42.391 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: **********************728 ms) 2024-05-15T12:38:42.391 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-05-15T12:38:42.391 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5708 ms) 2024-05-15T12:38:42.391 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-05-15T12:38:42.391 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (9080 ms) 2024-05-15T12:38:42.391 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-05-15T12:38:42.391 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2722 ms) 2024-05-15T12:38:42.391 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (3794 ms) 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (1361 ms) 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (496 ms) 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (394 ms) 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (932 ms) 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (315 ms) 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (228 ms) 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-05-15T12:38:42.392 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (154 ms) 2024-05-15T12:38:42.393 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-05-15T12:38:42.393 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (278 ms) 2024-05-15T12:38:42.393 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (173226 ms total) 2024-05-15T12:38:42.393 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: 2024-05-15T12:38:42.393 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [----------] Global test environment tear-down 2024-05-15T12:38:42.393 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (254412 ms total) 2024-05-15T12:38:42.393 INFO:tasks.workunit.client.0.smithi049.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-05-15T12:38:45.864 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: Running main() from gmock_main.cc 2024-05-15T12:38:45.864 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-05-15T12:38:45.864 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [----------] Global test environment set-up. 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (23064 ms) 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: no timeout :/ 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: no timeout :/ 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: no timeout :/ 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: no timeout :/ 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: no timeout :/ 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (638 ms) 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (738 ms) 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.Checksum (53 ms) 2024-05-15T12:38:45.865 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (107 ms) 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (130 ms) 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (126 ms) 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (24 ms) 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.TruncTest (72 ms) 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (2929 ms) 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (418 ms) 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-05-15T12:38:45.866 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.RmXattr (358 ms) 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIo.XattrIter (91 ms) 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [----------] 14 tests from LibRadosIo (28748 ms total) 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (93616 ms) 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (55430 ms) 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (8610 ms) 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (4325 ms) 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (8070 ms) 2024-05-15T12:38:45.867 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (332 ms) 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (326 ms) 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (365 ms) 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (955 ms) 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (407 ms) 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [----------] 10 tests from LibRadosIoEC (172436 ms total) 2024-05-15T12:38:45.868 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: 2024-05-15T12:38:45.869 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [----------] Global test environment tear-down 2024-05-15T12:38:45.869 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [==========] 24 tests from 2 test suites ran. (257898 ms total) 2024-05-15T12:38:45.869 INFO:tasks.workunit.client.0.smithi049.stdout: api_io: [ PASSED ] 24 tests. 2024-05-15T12:38:52.198 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:07:21.406 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:52.201 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:07:21.410 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:52.371 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:07:21.580 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:52.904 INFO:tasks.workunit.client.0.smithi049.stdout:********* 2024-05-15T12:38:52.904 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_9_[63401]: starting. 2024-05-15T12:38:52.904 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_9_[63401]: creating pool ceph_test_rados_list_parallel.smithi049-40671 2024-05-15T12:38:52.904 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_9_[63401]: created object 0... 2024-05-15T12:38:52.904 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_9_[63401]: created object 25... 2024-05-15T12:38:52.904 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_9_[63401]: created object 49... 2024-05-15T12:38:52.904 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_9_[63401]: finishing. 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_9_[63401]: shutting down. 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_10_[63402]: starting. 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_10_[63402]: listing objects. 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_10_[63402]: listed object 0... 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_10_[63402]: listed object 25... 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_10_[63402]: listed object 50... 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_10_[63402]: listed object 75... 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_10_[63402]: saw 100 objects 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_10_[63402]: shutting down. 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.905 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_13_[63405]: starting. 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_13_[63405]: removed 25 objects... 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_13_[63405]: removed half of the objects 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_13_[63405]: removed 50 objects... 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_13_[63405]: removed 50 objects 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_13_[63405]: shutting down. 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_11_[63403]: starting. 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_11_[63403]: added 25 objects... 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_11_[63403]: added half of the objects 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_11_[63403]: added 50 objects... 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_11_[63403]: added 50 objects 2024-05-15T12:38:52.906 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_11_[63403]: shutting down. 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_12_[63404]: starting. 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_12_[63404]: added 25 objects... 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_12_[63404]: added half of the objects 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_12_[63404]: added 50 objects... 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_12_[63404]: added 50 objects 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_12_[63404]: shutting down. 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.907 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_14_[72973]: starting. 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_14_[72973]: creating pool ceph_test_rados_list_parallel.smithi049-40671 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_14_[72973]: created object 0... 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_14_[72973]: created object 25... 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_14_[72973]: created object 49... 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_14_[72973]: finishing. 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_14_[72973]: shutting down. 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_15_[72974]: starting. 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_15_[72974]: listing objects. 2024-05-15T12:38:52.908 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_15_[72974]: listed object 0... 2024-05-15T12:38:53.073 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:07:22.240 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:53.093 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:07:22.261 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:53.149 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:07:22.358 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:53.173 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:07:22.378 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:53.259 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:07:22.426 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:38:53.487 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:07:22.696 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:02.322 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_15_[72974]: listed object : flushing 2024-05-15T12:39:02.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: flushed 2024-05-15T12:39:02.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3191 ms) 2024-05-15T12:39:02.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-05-15T12:39:02.322 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: trying... 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: handle_notify cookie 94268382500160 notify_id 816043786246 notifier_gid 4846 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: timed out 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: flushing 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: flushed 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3207 ms) 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: List watches 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: notify2 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: handle_notify cookie 94268382500160 notify_id 820338753541 notifier_gid 4846 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: notify2 done 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: watch_check 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: unwatch2 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: flushing 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: done 2024-05-15T12:39:02.323 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (4620 ms) 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: List watches 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: notify2 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: handle_notify cookie 94268382500160 notify_id 824633720838 notifier_gid 4846 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: notify2 done 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: watch_check 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: unwatch2 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: flushing 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: done 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3635 ms) 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (185036 ms total) 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (274113 ms total) 2024-05-15T12:39:02.324 INFO:tasks.workunit.client.0.smithi049.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-05-15T12:39:06.748 INFO:tasks.workunit.client.0.smithi049.stdout:25... 2024-05-15T12:39:06.748 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_15_[72974]: listed object 50... 2024-05-15T12:39:06.748 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_15_[72974]: listed object 75... 2024-05-15T12:39:06.748 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_15_[72974]: listed object 100... 2024-05-15T12:39:06.748 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_15_[72974]: listed object 125... 2024-05-15T12:39:06.748 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_15_[72974]: saw 150 objects 2024-05-15T12:39:06.748 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_15_[72974]: shutting down. 2024-05-15T12:39:06.748 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:39:06.748 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_16_[72975]: starting. 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_16_[72975]: added 25 objects... 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_16_[72975]: added half of the objects 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_16_[72975]: added 50 objects... 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_16_[72975]: added 50 objects 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: process_16_[72975]: shutting down. 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******************************* 2024-05-15T12:39:06.749 INFO:tasks.workunit.client.0.smithi049.stdout: list_parallel: ******* SUCCESS ********** 2024-05-15T12:39:19.951 INFO:tasks.workunit.client.0.smithi049.stdout:0x55fdb6c51530 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c51640 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7584bb0 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c4f380 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7584cc0 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7584df0 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75833b0 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7584f40 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7582f00 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75850f0 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7585240 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7585aa0 2024-05-15T12:39:19.952 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7586100 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7586740 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7586d20 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7586ed0 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7586fe0 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75870f0 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7583ff0 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7584610 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7584780 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7586290 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7585370 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75863a0 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75864b0 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7586600 2024-05-15T12:39:19.953 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75835d0 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7583b40 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75836e0 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7583830 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75803f0 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75855f0 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7583010 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7583120 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7583230 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7583960 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7583d90 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75848b0 2024-05-15T12:39:19.954 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7582a50 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7585700 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7585810 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7585920 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7584140 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7582b60 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7582c70 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7582da0 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c41600 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb757f5e0 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb757ff80 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb75800d0 2024-05-15T12:39:19.955 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb7580200 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c41710 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c41880 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: waiting 0x55fdb6c41a30 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: done. 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (104010 ms) 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (104010 ms total) 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMisc.Exec (93034 ms) 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (4872 ms) 2024-05-15T12:39:19.956 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-05-15T12:39:19.957 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3456 ms) 2024-05-15T12:39:19.957 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-05-15T12:39:19.957 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5471 ms) 2024-05-15T12:39:19.957 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-05-15T12:39:19.957 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-05-15T12:39:19.957 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-05-15T12:39:19.957 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-05-15T12:39:19.957 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-05-15T12:39:19.957 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (75209 ms) 2024-05-15T12:39:19.961 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [----------] 8 tests from LibRadosMisc (182042 ms total) 2024-05-15T12:39:19.961 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: 2024-05-15T12:39:19.961 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [----------] Global test environment tear-down 2024-05-15T12:39:19.961 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (291904 ms total) 2024-05-15T12:39:19.961 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc: [ PASSED ] 12 tests. 2024-05-15T12:39:23.710 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:07:52.919 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:23.740 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:07:52.949 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:23.747 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:07:52.956 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:23.752 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:07:52.961 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:23.802 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:07:52.969 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:23.830 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:07:52.998 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:23.896 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:07:53.064 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:23.958 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:07:53.167 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:24.011 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:07:53.220 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:44.021 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:13.230 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:44.082 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:13.290 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:44.101 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:13.268 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:44.117 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:13.285 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:44.197 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:13.406 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:44.208 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:13.376 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:44.228 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:13.437 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:44.238 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:13.447 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:44.332 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:13.541 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:39:53.172 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: Running main() from gmock_main.cc 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [----------] Global test environment set-up. 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (42207 ms) 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (46255 ms) 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (70209 ms) 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (8213 ms) 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (166885 ms total) 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-05-15T12:39:53.173 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (9271 ms) 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (7371 ms) 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (13924 ms) 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (30566 ms total) 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (30221 ms) 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (22390 ms) 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (24005 ms) 2024-05-15T12:39:53.174 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (4427 ms) 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (81043 ms total) 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (15324 ms) 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (8645 ms) 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (23969 ms total) 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [----------] Global test environment tear-down 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (325075 ms total) 2024-05-15T12:39:53.175 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-05-15T12:39:53.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-05-15T12:39:53.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-05-15T12:39:53.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-05-15T12:39:53.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-05-15T12:39:53.197 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: seed 40327 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (39767 ms) 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (8740 ms) 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (13634 ms) 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (18053 ms) 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (13112 ms) 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (93306 ms total) 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: 2024-05-15T12:39:53.198 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (24113 ms) 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (8712 ms) 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (68110 ms) 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (33241 ms) 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (24613 ms) 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1882-gae248125/rpm/el9/BUILD/ceph-19.0.0-1882-gae248125/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-05-15T12:39:53.199 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi049-40327-7 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: waiting for snaps to purge 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (41226 ms) 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (200016 ms total) 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (5 ms) 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5553 ms) 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5558 ms total) 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-05-15T12:39:53.200 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-05-15T12:39:53.201 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (5118 ms) 2024-05-15T12:39:53.201 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-05-15T12:40:04.031 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] read_operations: Running main() from gmock_main.cc 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [----------] Global test environment set-up. 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (43999 ms) 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (36787 ms) 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (12645 ms) 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (32147 ms) 2024-05-15T12:40:04.032 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (75985 ms) 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (10844 ms) 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (18912 ms) 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (37060 ms) 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (24032 ms) 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (6541 ms) 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-05-15T12:40:04.033 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (16514 ms) 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (9193 ms) 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (5870 ms) 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (4985 ms) 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (335515 ms total) 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [----------] Global test environment tear-down 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (335515 ms total) 2024-05-15T12:40:04.034 INFO:tasks.workunit.client.0.smithi049.stdout: read_operations: [ PASSED ] 14 tests. 2024-05-15T12:40:04.138 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:33.347 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:04.143 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:33.352 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:04.212 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:33.380 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:04.232 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:33.441 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:04.235 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:33.444 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:04.240 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:33.408 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:04.294 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:33.503 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:04.314 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:33.482 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:04.320 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:33.529 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:08.622 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: Running main() from gmock_main.cc 2024-05-15T12:40:08.622 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-05-15T12:40:08.622 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [----------] Global test environment set-up. 2024-05-15T12:40:08.622 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-05-15T12:40:08.622 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-05-15T12:40:08.622 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (42834 ms) 2024-05-15T12:40:08.622 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-05-15T12:40:08.622 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (55420 ms) 2024-05-15T12:40:08.622 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (16363 ms) 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (49006 ms) 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (35737 ms) 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (199360 ms total) 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (14277 ms) 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (50243 ms) 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (37468 ms) 2024-05-15T12:40:08.623 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (9473 ms) 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (7860 ms) 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi049-41184-11. 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: Waiting for snaps to purge. 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (21410 ms) 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (140731 ms total) 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [----------] Global test environment tear-down 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (340091 ms total) 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: snapshots: [ PASSED ] 11 tests. 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: io: Running main() from gmock_main.cc 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-05-15T12:40:08.624 INFO:tasks.workunit.client.0.smithi049.stdout: io: [----------] Global test environment set-up. 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [----------] 14 tests from NeoRadosIo 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.Limits (37541 ms) 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (70542 ms) 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.ReadOp (92460 ms) 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.SparseRead (10916 ms) 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.RoundTrip (16169 ms) 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (31569 ms) 2024-05-15T12:40:08.625 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (29541 ms) 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (16227 ms) 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (6998 ms) 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.Trunc (6812 ms) 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.Remove (6777 ms) 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (4657 ms) 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-05-15T12:40:08.626 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.RmXattr (4995 ms) 2024-05-15T12:40:08.627 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-05-15T12:40:08.627 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ OK ] NeoRadosIo.GetXattrs (4979 ms) 2024-05-15T12:40:08.627 INFO:tasks.workunit.client.0.smithi049.stdout: io: [----------] 14 tests from NeoRadosIo (340183 ms total) 2024-05-15T12:40:08.627 INFO:tasks.workunit.client.0.smithi049.stdout: io: 2024-05-15T12:40:08.627 INFO:tasks.workunit.client.0.smithi049.stdout: io: [----------] Global test environment tear-down 2024-05-15T12:40:08.627 INFO:tasks.workunit.client.0.smithi049.stdout: io: [==========] 14 tests from 1 test suite ran. (340183 ms total) 2024-05-15T12:40:08.627 INFO:tasks.workunit.client.0.smithi049.stdout: io: [ PASSED ] 14 tests. 2024-05-15T12:40:15.424 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:44.633 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:15.430 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:44.639 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:15.438 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:44.606 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:15.440 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:44.608 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:15.457 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:44.666 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:15.461 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:44.670 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:15.568 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:44.736 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:15.584 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:44.792 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:15.608 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:44.817 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:24.139 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: Running main() from gmock_main.cc 2024-05-15T12:40:24.139 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-05-15T12:40:24.139 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [----------] Global test environment set-up. 2024-05-15T12:40:24.139 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-05-15T12:40:24.139 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-05-15T12:40:24.139 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (41287 ms) 2024-05-15T12:40:24.139 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-05-15T12:40:24.139 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (72018 ms) 2024-05-15T12:40:24.139 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-05-15T12:40:24.139 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (17972 ms) 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (15616 ms) 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (11644 ms) 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (12192 ms) 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (184873 ms) 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (355602 ms total) 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [----------] Global test environment tear-down 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (355602 ms total) 2024-05-15T12:40:24.140 INFO:tasks.workunit.client.0.smithi049.stdout: write_operations: [ PASSED ] 7 tests. 2024-05-15T12:40:24.159 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:53.368 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:24.242 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:53.450 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:24.279 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:53.446 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:24.281 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:08:53.490 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:24.290 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:53.457 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:24.297 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:53.506 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:24.303 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:53.512 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:24.319 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:08:53.486 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:24.339 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:08:53.548 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:31.234 INFO:tasks.workunit.client.0.smithi049.stdout:LibRadosSnapshotsECPP.SnapRemovePP (4113 ms) 2024-05-15T12:40:31.234 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-05-15T12:40:31.234 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (5441 ms) 2024-05-15T12:40:31.234 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-05-15T12:40:31.234 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (4004 ms) 2024-05-15T12:40:31.234 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (18676 ms total) 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (7940 ms) 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (7649 ms) 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4419 ms) 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (20008 ms total) 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (363100 ms total) 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-05-15T12:40:31.235 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-05-15T12:40:31.236 INFO:tasks.workunit.client.0.smithi049.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-05-15T12:40:32.260 INFO:tasks.workunit.client.0.smithi049.stdout: list: Running main() from gmock_main.cc 2024-05-15T12:40:32.260 INFO:tasks.workunit.client.0.smithi049.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-05-15T12:40:32.260 INFO:tasks.workunit.client.0.smithi049.stdout: list: [----------] Global test environment set-up. 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [----------] 3 tests from NeoradosList 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [ OK ] NeoradosList.ListObjects (43063 ms) 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [ OK ] NeoradosList.ListObjectsNS (189507 ms) 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [ OK ] NeoradosList.ListObjectsMany (131236 ms) 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [----------] 3 tests from NeoradosList (363806 ms total) 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [----------] Global test environment tear-down 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [==========] 3 tests from 1 test suite ran. (363806 ms total) 2024-05-15T12:40:32.261 INFO:tasks.workunit.client.0.smithi049.stdout: list: [ PASSED ] 3 tests. 2024-05-15T12:40:38.974 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-05-15T12:40:38.974 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-05-15T12:40:38.974 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (18582 ms) 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (30137 ms) 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (45061 ms) 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (36550 ms) 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-05-15T12:40:38.975 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (6262 ms) 2024-05-15T12:40:38.976 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-05-15T12:40:38.976 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (43665 ms) 2024-05-15T12:40:38.976 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-05-15T12:40:38.976 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (190453 ms) 2024-05-15T12:40:38.976 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (370710 ms total) 2024-05-15T12:40:38.976 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: 2024-05-15T12:40:38.976 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-05-15T12:40:38.976 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (370710 ms total) 2024-05-15T12:40:38.976 INFO:tasks.workunit.client.0.smithi049.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-05-15T12:40:44.352 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:09:13.561 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:44.356 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:09:13.564 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:44.402 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:09:13.569 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:44.406 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:09:13.574 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:44.412 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:09:13.621 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:44.436 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:09:13.645 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:44.442 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:09:13.651 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:44.455 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:09:13.622 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:44.502 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:09:13.710 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:40:55.152 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=9 expected=9 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:de5d7c5f:::12:head 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:de5d7c5f:::12:head expected=19:de5d7c5f:::12:head 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:de5d7c5f:::12:head -> 12 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=12 expected=12 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:d83876eb:::4:head 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:d83876eb:::4:head expected=19:d83876eb:::4:head 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:d83876eb:::4:head -> 4 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=4 expected=4 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:6cac518f:::0:head 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:6cac518f:::0:head expected=19:6cac518f:::0:head 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:6cac518f:::0:head -> 0 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=0 expected=0 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:c4fdafeb:::6:head 2024-05-15T12:40:55.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:c4fdafeb:::6:head expected=19:c4fdafeb:::6:head 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:c4fdafeb:::6:head -> 6 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=6 expected=6 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:bd63b0f1:::8:head 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:bd63b0f1:::8:head expected=19:bd63b0f1:::8:head 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:bd63b0f1:::8:head -> 8 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=8 expected=8 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:b29083e3:::5:head 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:b29083e3:::5:head expected=19:b29083e3:::5:head 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:b29083e3:::5:head -> 5 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=5 expected=5 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:89d3ae78:::11:head 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:89d3ae78:::11:head expected=19:89d3ae78:::11:head 2024-05-15T12:40:55.154 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:89d3ae78:::11:head -> 11 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=11 expected=11 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : seek to 19:02547ec2:::1:head 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : cursor()=19:02547ec2:::1:head expected=19:02547ec2:::1:head 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: > 19:02547ec2:::1:head -> 1 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: : entry=1 expected=1 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (17833 ms) 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (126898 ms) 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: split 0/5 -> MIN 19:33333333::::head 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: split 1/5 -> 19:33333333::::head 19:66666666::::head 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: split 2/5 -> 19:66666666::::head 19:99999999::::head 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: split 3/5 -> 19:99999999::::head 19:cccccccc::::head 2024-05-15T12:40:55.155 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: split 4/5 -> 19:cccccccc::::head MAX 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (9015 ms) 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [----------] 7 tests from LibRadosList (370870 ms total) 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1989 ms) 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: myset foo1,foo2,foo3 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo1 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo2 2024-05-15T12:40:55.156 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo3 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: myset foo1,foo4,foo5 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo4 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo5 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo1 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: myset foo6,foo7 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo7 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: foo6 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: ns1:foo4 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: ns1:foo5 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: ns2:foo7 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: ns2:foo6 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: ns1:foo1 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: :foo1 2024-05-15T12:40:55.157 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: :foo2 2024-05-15T12:40:55.158 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: :foo3 2024-05-15T12:40:55.158 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (2156 ms) 2024-05-15T12:41:02.548 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] Li api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-05-15T12:41:02.548 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-05-15T12:41:02.548 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: seed 40176 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (5 ms) 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (20567 ms) 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (2362 ms) 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (199 ms) 2024-05-15T12:41:02.549 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (227 ms) 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (61 ms) 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (76 ms) 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (213 ms) 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (71 ms) 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (5446 ms) 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-05-15T12:41:02.550 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (168 ms) 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (112 ms) 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (324 ms) 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: osd_max_attr_size = 0 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (248171 ms) 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (15940 ms) 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: done waiting, doing copies 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: waiting for final deep scrubs... 2024-05-15T12:41:02.551 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: done waiting 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (88989 ms) 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (199 ms) 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (76 ms) 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4711 ms) 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-05-15T12:41:02.552 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-05-15T12:41:02.701 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 22bRadosListEC.ListObjectsStart 2024-05-15T12:41:02.701 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 1 0 2024-05-15T12:41:02.701 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 10 0 2024-05-15T12:41:02.701 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 13 0 2024-05-15T12:41:02.701 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 7 0 2024-05-15T12:41:02.701 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 14 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 0 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 15 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 11 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 5 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 8 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 6 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 3 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 4 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 12 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 9 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 2 0 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-05-15T12:41:02.702 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (2130 ms) 2024-05-15T12:41:02.703 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [----------] 3 tests from LibRadosListEC (6275 ms total) 2024-05-15T12:41:02.703 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 2024-05-15T12:41:02.703 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-05-15T12:41:02.703 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-05-15T12:41:02.703 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3391 ms) 2024-05-15T12:41:02.703 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [----------] 1 test from LibRadosListNP (3391 ms total) 2024-05-15T12:41:02.703 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: 2024-05-15T12:41:02.703 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [----------] Global test environment tear-down 2024-05-15T12:41:02.703 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [==========] 11 tests from 3 test suites ran. (394688 ms total) 2024-05-15T12:41:02.703 INFO:tasks.workunit.client.0.smithi049.stdout: api_list: [ PASSED ] 11 tests. 2024-05-15T12:41:04.296 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:09:33.505 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:04.301 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:09:33.510 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:04.305 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:09:33.473 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:04.309 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:09:33.476 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:04.360 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:09:33.527 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:04.361 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:09:33.570 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:04.373 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:09:33.582 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:04.375 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:09:33.584 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:04.423 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:09:33.631 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:13.579 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: Running main() from gmock_main.cc 2024-05-15T12:41:13.579 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-05-15T12:41:13.579 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [----------] Global test environment set-up. 2024-05-15T12:41:13.579 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-05-15T12:41:13.579 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-05-15T12:41:13.579 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (48176 ms) 2024-05-15T12:41:13.579 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-05-15T12:41:13.579 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (203164 ms) 2024-05-15T12:41:13.580 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-05-15T12:41:13.580 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (153755 ms) 2024-05-15T12:41:13.580 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [----------] 3 tests from NeoradosECList (405095 ms total) 2024-05-15T12:41:13.580 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: 2024-05-15T12:41:13.580 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [----------] Global test environment tear-down 2024-05-15T12:41:13.580 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (405095 ms total) 2024-05-15T12:41:13.580 INFO:tasks.workunit.client.0.smithi049.stdout: ec_list: [ PASSED ] 3 tests. 2024-05-15T12:41:24.393 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:09:53.602 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:24.401 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:09:53.610 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:24.446 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:09:53.613 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:24.447 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:09:53.615 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:24.451 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:09:53.660 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:24.454 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:09:53.663 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:24.464 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:09:53.673 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:24.492 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:09:53.659 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:24.504 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:09:53.713 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:35.181 INFO:tasks.workunit.client.0.smithi049.stdout: tests from LibRadosMiscPP (387917 ms total) 2024-05-15T12:41:35.181 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: 2024-05-15T12:41:35.181 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-05-15T12:41:35.181 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-05-15T12:41:35.181 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (4582 ms) 2024-05-15T12:41:35.181 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (4582 ms total) 2024-05-15T12:41:35.181 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: 2024-05-15T12:41:35.181 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-05-15T12:41:35.181 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (468 ms) 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (58 ms) 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (526 ms total) 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (497 ms) 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (65 ms) 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (562 ms total) 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: 2024-05-15T12:41:35.182 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (601 ms) 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (41 ms) 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (642 ms total) 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1572 ms) 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1572 ms total) 2024-05-15T12:41:35.183 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: 2024-05-15T12:41:35.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-05-15T12:41:35.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (427094 ms total) 2024-05-15T12:41:35.184 INFO:tasks.workunit.client.0.smithi049.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-05-15T12:41:44.674 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:10:13.882 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:44.704 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:10:13.913 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:44.714 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:10:13.882 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:44.720 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:10:13.887 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:44.723 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:10:13.931 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:44.727 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:10:13.936 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:44.746 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:10:13.954 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:44.783 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:10:13.950 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:41:44.794 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:10:14.003 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:04.664 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:10:33.872 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:04.666 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:10:33.875 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:04.689 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:10:33.897 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:04.718 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:10:33.886 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:04.719 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:10:33.928 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:04.720 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:10:33.887 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:04.720 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:10:33.929 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:04.741 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:10:33.908 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:04.742 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:10:33.951 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:24.897 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:10:54.106 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:24.961 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:10:54.170 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:25.003 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:10:54.212 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:25.016 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:10:54.183 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:25.018 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:10:54.185 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:25.025 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:10:54.234 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:25.027 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:10:54.236 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:25.062 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:10:54.230 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:25.063 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:10:54.271 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:39.878 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:09.087 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:39.882 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:09.091 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:39.928 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:09.136 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:39.935 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:09.102 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:39.939 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:09.106 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:39.944 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:09.153 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:39.949 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:09.158 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:39.979 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:09.146 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:39.992 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:09.201 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:44.885 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:14.093 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:44.887 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:14.096 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:44.907 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:14.116 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:44.929 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:14.097 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:44.931 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:14.098 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:44.949 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:14.116 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:44.950 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:14.158 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:44.953 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:14.162 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:42:44.970 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:14.179 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:04.899 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:34.107 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:04.948 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:34.115 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:04.949 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:34.116 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:04.973 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:34.181 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:04.975 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:34.184 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:04.980 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:34.189 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:04.990 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:34.157 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:05.038 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:34.246 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:05.042 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:34.251 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:25.109 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:54.317 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:25.112 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:54.320 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:25.150 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:11:54.358 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:25.180 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:54.388 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:25.181 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:54.390 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:25.181 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:54.349 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:25.183 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:54.350 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:25.222 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:11:54.389 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:25.224 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:11:54.433 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:45.275 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:12:14.484 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:45.281 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:12:14.490 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:45.335 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:12:14.543 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:45.346 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:12:14.513 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:45.348 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:12:14.515 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:45.356 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:12:14.565 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:45.359 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:12:14.568 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:45.396 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:12:14.563 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:43:45.420 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:12:14.628 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:05.184 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:12:34.392 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:05.186 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:12:34.394 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:05.209 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:12:34.418 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:05.236 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:12:34.403 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:05.237 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:12:34.404 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:05.238 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:12:34.447 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:05.240 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:12:34.449 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:05.260 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:12:34.427 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:05.263 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:12:34.471 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:30.537 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:12:59.746 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:30.542 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:12:59.751 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:30.611 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:12:59.778 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:30.613 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:12:59.780 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:30.617 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:12:59.825 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:30.635 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:12:59.844 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:30.645 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:12:59.853 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:30.669 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:12:59.836 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:30.709 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:12:59.918 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:31.457 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:00.624 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:31.458 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:00.626 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:31.470 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:00.638 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:31.644 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:00.853 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:31.645 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:00.854 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:31.655 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:00.864 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:31.684 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:00.893 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:31.685 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:00.894 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:31.695 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:00.904 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:40.583 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:09.791 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:40.587 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:09.796 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:40.615 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:09.824 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:40.636 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:09.803 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:40.640 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:09.807 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:40.642 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:09.850 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:40.649 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:09.858 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:40.663 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:09.830 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:40.677 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:09.886 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:50.607 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:19.815 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:50.609 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:19.818 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:50.634 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:19.843 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:50.657 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:19.824 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:50.660 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:19.827 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:50.669 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:19.878 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:50.675 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:19.884 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:50.683 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:19.850 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:50.705 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:19.913 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:54.856 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:24.064 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:54.870 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:24.079 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:54.884 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:24.051 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:54.887 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:24.055 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:54.895 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:24.103 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:54.964 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:24.172 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:54.969 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:24.136 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:54.984 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:24.193 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:54.992 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:24.200 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:55.572 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:24.780 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:55.618 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:24.785 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:55.621 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:24.830 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:55.625 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:24.833 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:55.631 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:24.839 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:55.671 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:24.838 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:55.678 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:24.887 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:55.681 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:24.848 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:44:55.689 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:24.898 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:10.845 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:40.054 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:10.848 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:40.057 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:10.896 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:40.105 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:10.902 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:40.111 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:10.904 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:40.112 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:10.910 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:40.078 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:10.915 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:40.082 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:10.941 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:40.150 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:10.964 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:40.131 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:21.441 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-05-15T12:45:21.441 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: seed 40250 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (21282 ms) 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (94121 ms) 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (17229 ms) 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (132632 ms total) 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (89743 ms) 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-05-15T12:45:21.442 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (56640 ms) 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (46843 ms) 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: my_snaps [3] 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: my_snaps [4,3] 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: my_snaps [5,4,3] 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: promoting some heads 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: promoting from clones for snap 6 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: promoting from clones for snap 5 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: promoting from clones for snap 4 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: promoting from clones for snap 3 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: waiting for scrubs... 2024-05-15T12:45:21.443 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: done waiting 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (140150 ms) 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (12086 ms) 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8761 ms) 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8573 ms) 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10097 ms) 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (12073 ms) 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10905 ms) 2024-05-15T12:45:21.444 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (12101 ms) 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (40085 ms) 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10932 ms) 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (11814 ms) 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (16595 ms) 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8840 ms) 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-05-15T12:45:21.445 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (14369 ms) 2024-05-15T12:45:25.747 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:54.955 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:25.751 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:54.959 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:25.794 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:13:55.002 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:25.796 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:54.963 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:25.799 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:54.966 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:25.804 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:55.013 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:25.807 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:55.016 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:25.827 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:13:54.994 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:25.853 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:13:55.061 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:30.972 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:00.180 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:30.974 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:00.183 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:31.000 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:00.209 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:31.022 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:00.189 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:31.027 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:00.194 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:31.035 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:00.244 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:31.040 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:00.249 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:31.046 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:00.213 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:31.057 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:00.266 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:41.045 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:10.254 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:41.050 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:10.258 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:41.082 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:10.290 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:41.099 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:10.266 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:41.107 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:10.274 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:41.108 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:10.316 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:41.111 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:10.320 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:41.135 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:10.302 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:41.149 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:10.358 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:46.138 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-05-15T12:45:46.138 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: hmm, no HitSet yet 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: ok, hit_set contains 170:602f83fe:::foo:head 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (11606 ms) 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg_num = 32 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 0 ls 1715777140,0 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 1 ls 1715777140,0 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 2 ls 1715777140,0 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 3 ls 1715777140,0 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 4 ls 1715777140,0 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 5 ls 1715777140,0 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 6 ls 1715777141,0 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 7 ls 1715777140,0 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 8 ls 1715777140,0 2024-05-15T12:45:46.139 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 9 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 10 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 11 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 12 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 13 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 14 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 15 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 16 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 17 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 18 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 19 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 20 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 21 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 22 ls 1715777140,0 2024-05-15T12:45:46.140 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 23 ls 1715777140,0 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 24 ls 1715777140,0 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 25 ls 1715777140,0 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 26 ls 1715777140,0 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 27 ls 1715777140,0 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 28 ls 1715777140,0 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 29 ls 1715777140,0 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 30 ls 1715777140,0 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg 31 ls 1715777140,0 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: pg_num = 32 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:6cac518f:::0:head 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:02547ec2:::1:head 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:f905c69b:::2:head 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:cfc208b3:::3:head 2024-05-15T12:45:46.141 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d83876eb:::4:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:b29083e3:::5:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c4fdafeb:::6:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:5c6b0b28:::7:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:bd63b0f1:::8:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:e960b815:::9:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:52ea6a34:::10:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:89d3ae78:::11:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:de5d7c5f:::12:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:566253c9:::13:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:62a1935d:::14:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:863748b0:::15:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:3958e169:::16:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:4d4dabf9:::17:head 2024-05-15T12:45:46.142 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:8391935d:::18:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:28883081:::19:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:69259c59:::20:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:4bdb80b7:::21:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:a11c5d71:::22:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:271af37b:::23:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:95b121be:::24:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:58d1031b:::25:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:0a050783:::26:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c709704c:::27:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:cbe56eaf:::28:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:86b4b162:::29:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:70d89383:::30:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:dd450c7c:::31:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:6d5729b1:::32:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c388f3fb:::33:head 2024-05-15T12:45:46.143 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:56cfea31:::34:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:9dbc1bf7:::35:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:40b74ccd:::36:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:4d5aaf42:::37:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:920f362c:::38:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:6cc53222:::39:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:9cad833f:::40:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:1ea84d41:::41:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c4480ef6:::42:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:a694361e:::43:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d1bd33e9:::44:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:ddc2cd5d:::45:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:2b782207:::46:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:7b187fca:::47:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:90ecdf6f:::48:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:a5ed95fe:::49:head 2024-05-15T12:45:46.144 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:ea0eaa55:::50:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:f33ef17b:::51:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:a0d1b2f6:::52:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:60c5229e:::53:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:edcbc575:::54:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:102cf253:::55:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:efb7fb0b:::56:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:50d0a326:::57:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d4dc5daf:::58:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:3a130462:::59:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:ec87ed71:::60:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d5bc9454:::61:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:3ddfe313:::62:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:7c2816b9:::63:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:47e00e4d:::64:head 2024-05-15T12:45:46.145 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c6410c18:::65:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:b48ed237:::66:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:cd63ad31:::67:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:b179e92b:::68:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:0d9f741a:::69:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:6d3352ae:::70:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c6d5c19e:::71:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:bc4729c3:::72:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:77e930b9:::73:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:0abeecfd:::74:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:b7c37e15:::75:head 2024-05-15T12:45:46.146 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:b6378398:::76:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:02bd68de:::77:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:cc795d2d:::78:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:630d4fea:::79:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:e0d29ef5:::80:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:fd6f13d2:::81:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:606461d5:::82:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:eadbdc43:::83:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:8761d0bb:::84:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:9ef0186f:::85:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:e0d41294:::86:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:961de695:::87:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:1423148f:::88:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:633a8fa2:::89:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:a8653809:::90:head 2024-05-15T12:45:46.147 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:3dac8b33:::91:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:35aad435:::92:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:f6dcc343:::93:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:dbbdad87:::94:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:1cb48ce0:::95:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:03cd461c:::96:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:17a4ea99:::97:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:9993c9a7:::98:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:6394211c:::99:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:94c7ae57:::100:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:6fdee5bb:::101:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:9a477fd1:::102:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:eb850916:::103:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:affc56b9:::104:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:b42dc814:::105:head 2024-05-15T12:45:46.148 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:f319f8f0:::106:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:9a40b9de:::107:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:8b524f28:::108:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:e3de589f:::109:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:90f90a5b:::110:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:a7b4f1d7:::111:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:af51766e:::112:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:b6f90bd1:::113:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:e0261208:::114:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c9569ef7:::115:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:61bebe50:::116:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:fe93412b:::117:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d3d38bee:::118:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:3100ba0c:::119:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d0560ada:::120:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:f0ea8b35:::121:head 2024-05-15T12:45:46.149 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:766f231a:::122:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:a07a2582:::123:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:bd7c6b3a:::124:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:fb2ddaff:::125:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:4408e1fe:::126:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:ee1df7a7:::127:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c3002909:::128:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:4f48ffa9:::129:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:edf38733:::130:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c08425c0:::131:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:5f902d98:::132:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:41ea2c93:::133:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:813cee13:::134:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:0131818d:::135:head 2024-05-15T12:45:46.150 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:26ba5a85:::136:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:381b8a5a:::137:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:28797e47:::138:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:bfca7f22:::139:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:36807075:::140:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:80b03975:::141:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:5c15709b:::142:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:f39ea15e:::143:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:ea992956:::144:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:48887b1c:::145:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:9f24a9dd:::146:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:987f100b:::147:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d2dd3581:::148:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:7fed1808:::149:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c80b70e9:::150:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:85ed90f9:::151:head 2024-05-15T12:45:46.151 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:36428b24:::152:head 2024-05-15T12:45:46.152 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d044c34a:::153:head 2024-05-15T12:45:46.152 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:7c18bf58:::154:head 2024-05-15T12:45:46.152 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d1c21232:::155:head 2024-05-15T12:45:46.152 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:a7a3c575:::156:head 2024-05-15T12:45:46.152 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:87da0633:::157:head 2024-05-15T12:45:46.152 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d5ac3822:::158:head 2024-05-15T12:45:46.152 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:3f20522d:::159:head 2024-05-15T12:45:46.152 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:6ca26563:::160:head 2024-05-15T12:45:46.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:532ce135:::161:head 2024-05-15T12:45:46.153 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:c78863e6:::162:head 2024-05-15T12:45:50.915 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:20.123 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:50.917 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:20.126 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:50.935 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:20.144 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:50.972 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:20.139 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:50.974 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:20.141 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:50.986 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:20.195 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:50.991 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:20.199 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:50.993 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:20.160 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:51.010 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:20.219 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:56.099 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:25.308 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:56.104 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:25.313 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:56.120 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:25.329 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:56.154 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:25.321 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:56.155 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:25.322 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:56.169 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:25.336 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:56.178 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:25.387 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:56.180 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:25.389 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:45:56.199 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:25.407 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:10.992 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:40.200 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:10.994 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:40.203 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.038 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:40.247 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.044 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:40.211 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.050 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:40.216 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.065 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:40.274 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.067 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:40.275 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.076 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:40.243 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.095 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:40.303 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.675 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:40.842 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.676 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:40.843 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.685 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:40.852 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.738 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:40.946 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.739 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:40.947 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.749 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:40.957 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.762 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:40.970 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.763 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:40.972 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:11.772 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:40.981 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:26.040 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:55.248 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:26.044 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:55.253 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:26.073 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:14:55.282 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:26.093 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:55.260 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:26.095 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:55.262 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:26.110 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:55.319 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:26.113 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:55.322 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:26.121 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:14:55.288 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:26.140 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:14:55.349 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:31.070 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:00.278 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:31.074 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:00.283 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:31.104 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:00.313 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:31.130 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:00.297 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:31.135 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:00.302 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:31.136 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:00.345 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:31.141 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:00.349 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:31.153 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:00.320 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:31.160 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:00.369 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.145 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:25.353 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.146 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:25.355 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.182 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:25.390 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.183 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:25.350 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.189 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:25.357 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.190 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:25.399 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.193 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:25.402 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.227 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:25.394 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.235 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:25.443 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.699 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:25.865 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.705 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:25.872 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.729 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:25.896 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.861 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:26.069 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.862 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:26.071 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.877 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:26.085 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.917 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:26.126 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.919 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:26.127 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:46:56.938 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:26.146 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:00.104 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:7462ddf6:::163:head 2024-05-15T12:47:00.104 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:5d165639:::164:head 2024-05-15T12:47:00.104 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:f43765fc:::165:head 2024-05-15T12:47:00.104 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:b4c720e9:::166:head 2024-05-15T12:47:00.104 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:e694b040:::167:head 2024-05-15T12:47:00.104 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:afa38db2:::168:head 2024-05-15T12:47:00.104 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:77ba9f53:::169:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:87495034:::170:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:7c96bf0e:::171:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:dbe346cc:::172:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:e943ec24:::173:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:f97a9c0c:::174:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:6f26e74d:::175:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:4f95e106:::176:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:0e6f2f8f:::177:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:05db05f1:::178:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:38a78d66:::179:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:d095610b:::180:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:a1a9d709:::181:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:1e5d39db:::182:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:f7df4fb9:::183:head 2024-05-15T12:47:00.105 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:03a7f161:::184:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:ba70721e:::185:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:28e5662d:::186:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:973d52de:::187:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:4303eb1c:::188:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:b990b48e:::189:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:29b8165b:::190:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:3547f197:::191:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:7e260936:::192:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:1abec7b1:::193:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:10fdda93:::194:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:15817eea:::195:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:770bab57:::196:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:ed9e13e7:::197:head 2024-05-15T12:47:00.106 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:71471a8f:::198:head 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: checking for 171:10fb1d02:::199:head 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22983 ms) 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,0 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: first is 1715777164 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,0 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,0 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,1715777166,1715777167,0 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,1715777166,1715777167,0 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,1715777166,1715777167,0 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,1715777166,1715777167,1715777170,1715777171,0 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,1715777166,1715777167,1715777170,1715777171,0 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,1715777166,1715777167,1715777170,1715777171,0 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,1715777166,1715777167,1715777170,1715777171,1715777173,1715777174,0 2024-05-15T12:47:00.107 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,1715777166,1715777167,1715777170,1715777171,1715777173,1715777174,0 2024-05-15T12:47:00.108 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777164,1715777166,1715777167,1715777170,1715777171,1715777173,1715777174,0 2024-05-15T12:47:00.108 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777167,1715777170,1715777171,1715777173,1715777174,1715777176,1715777177,0 2024-05-15T12:47:00.108 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: first now 1715777167, trimmed 2024-05-15T12:47:00.108 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (23538 ms) 2024-05-15T12:47:00.108 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-05-15T12:47:00.108 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: foo0 2024-05-15T12:47:00.108 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-05-15T12:47:00.108 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (17486 ms) 2024-05-15T12:47:00.108 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-05-15T12:47:11.293 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:40.501 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:11.297 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:40.505 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:11.344 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:40.552 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:11.358 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:40.525 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:11.359 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:40.526 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:11.388 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:40.597 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:11.390 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:40.598 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:11.395 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:40.562 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:11.429 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:40.638 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.514 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:45.722 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.517 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:45.726 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.537 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:45.746 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.566 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:45.733 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.568 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:45.735 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.580 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:45.789 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.582 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:45.791 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.585 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:45.752 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:16.600 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:45.808 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:26.445 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:55.653 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:26.451 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:55.660 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:26.483 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:15:55.691 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:26.514 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:55.681 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:26.516 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:55.683 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:26.521 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:55.729 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:26.533 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:55.742 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:26.539 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:15:55.706 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:26.562 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:15:55.770 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:36.423 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:05.631 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:36.431 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:05.639 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:36.465 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:05.673 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:36.490 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:05.657 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:36.495 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:05.662 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:36.505 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:05.714 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:36.512 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:05.721 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:36.525 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:05.692 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:36.541 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:05.750 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:41.558 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:10.767 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:41.567 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:10.734 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:41.569 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:10.778 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:41.570 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:10.778 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:41.613 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:10.780 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:41.629 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:10.796 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:41.643 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:10.852 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:41.678 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:10.886 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:41.682 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:10.891 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:56.532 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:25.740 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:56.534 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:25.743 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:56.569 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:25.778 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:56.580 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:25.788 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:56.584 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:25.793 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:56.585 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:25.752 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:56.587 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:25.754 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:56.620 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:25.828 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:56.620 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:25.787 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:57.397 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:26.564 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:57.398 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:26.565 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:57.409 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:26.576 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:57.425 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:26.633 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:57.426 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:26.635 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:57.442 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:26.650 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:57.457 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:26.665 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:57.458 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:26.667 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:47:57.469 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:26.677 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:16.635 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:45.843 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:16.688 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:45.855 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:16.689 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:45.897 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:16.691 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:45.858 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:16.714 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:16:45.923 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:16.717 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:16:45.884 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:16.747 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:45.956 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:16.751 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:45.960 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:16.780 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:16:45.989 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:31.694 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:00.902 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:31.699 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:00.908 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:31.737 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:00.946 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:31.750 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:00.959 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:31.758 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:00.925 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:31.760 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:00.969 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:31.760 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:00.927 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:31.791 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:01.000 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:31.803 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:00.970 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:36.709 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:05.917 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:36.712 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:05.921 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:36.728 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:05.936 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:36.760 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:05.927 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:36.762 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:05.971 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:36.764 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:05.931 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:36.764 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:05.973 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:36.779 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:05.946 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:36.789 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:05.997 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:37.379 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:06.546 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:37.380 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:06.547 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:37.389 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:06.556 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:37.398 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:06.607 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:37.399 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:06.608 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:37.408 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:06.616 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:37.408 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:06.617 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:37.409 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:06.618 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:37.417 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:06.626 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:51.755 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:20.963 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:51.757 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:20.965 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:51.781 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:20.989 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:51.803 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:20.970 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:51.805 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:20.972 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:51.811 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:21.019 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:51.812 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:21.021 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:51.828 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:20.995 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:51.836 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:21.045 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:56.925 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:26.133 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:56.928 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:26.137 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:56.958 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:26.167 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:56.975 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:26.142 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:56.979 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:26.146 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:56.983 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:26.192 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:56.989 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:26.197 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:56.994 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:26.161 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:48:57.010 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:26.218 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:11.850 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:41.059 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:11.852 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:41.061 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:11.881 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:41.089 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:11.900 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:41.067 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:11.901 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:41.068 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:11.906 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:41.115 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:11.907 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:41.116 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:11.927 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:41.094 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:11.936 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:41.145 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:16.975 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:46.183 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:16.979 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:46.188 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:16.993 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:17:46.201 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:17.030 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:46.197 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:17.031 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:46.199 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:17.044 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:17:46.211 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:17.061 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:46.270 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:17.062 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:46.271 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:17.075 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:17:46.283 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:31.962 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:01.171 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:31.964 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:01.173 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:31.986 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:01.195 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:32.017 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:01.184 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:32.018 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:01.227 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:32.018 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:01.186 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:32.019 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:01.228 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:32.042 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:01.250 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:32.043 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:01.210 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:41.988 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:11.196 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:41.994 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:11.202 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:42.013 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:11.222 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:42.041 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:11.208 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:42.043 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:11.210 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:42.043 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:11.252 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:42.046 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:11.255 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:42.059 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:11.226 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:42.064 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:11.272 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.110 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:21.319 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.117 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:21.325 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.149 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:21.357 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.162 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:21.329 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.164 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:21.331 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.189 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:21.356 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.195 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:21.403 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.197 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:21.405 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.230 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:21.438 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.856 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:22.023 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.857 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:22.024 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:52.872 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:22.039 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:53.112 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:22.320 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:53.113 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:22.322 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:53.129 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:22.337 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:53.130 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:22.339 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:53.143 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:22.352 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:49:53.153 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:22.361 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:02.106 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:31.314 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:02.107 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:31.316 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:02.135 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:31.343 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:02.159 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:31.326 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:02.160 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:31.328 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:02.162 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:31.370 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:02.163 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:31.372 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:02.186 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:31.353 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:02.189 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:31.397 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:07.119 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:36.327 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:07.123 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:36.332 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:07.135 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:36.344 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:07.169 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:36.336 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:07.171 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:36.339 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:07.177 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:36.386 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:07.179 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:36.388 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:07.183 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:36.350 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:07.191 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:36.399 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:22.151 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:51.359 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:22.155 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:51.364 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:22.173 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:51.381 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:22.207 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:51.373 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:22.208 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:51.375 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:22.224 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:51.392 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:22.237 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:51.445 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:22.240 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:51.449 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:22.258 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:51.467 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:22.785 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T12:50:22.782+0000 26292640 -1 snap_mapper.add_oid found existing snaps mapped on 189:06a9dcd5:test-rados-api-smithi049-40250-80::foo:2, removing 2024-05-15T12:50:22.807 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:50:22.806+0000 28a97640 -1 snap_mapper.add_oid found existing snaps mapped on 189:06a9dcd5:test-rados-api-smithi049-40250-80::foo:2, removing 2024-05-15T12:50:27.191 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:56.399 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:27.194 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:56.403 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:27.211 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:18:56.420 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:27.242 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:56.408 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:27.242 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:56.410 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:27.257 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:18:56.424 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:27.259 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:56.467 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:27.261 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:56.470 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:27.277 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:18:56.486 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.242 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:11.450 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.246 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:11.455 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.282 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:11.490 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.296 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:11.463 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.298 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:11.465 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.304 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:11.513 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.307 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:11.515 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.331 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:11.498 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.370 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:11.578 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.879 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:12.046 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.881 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:12.049 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.891 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:12.058 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.993 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:12.201 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:42.995 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:12.203 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.005 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:12.214 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.016 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:12.225 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.017 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:12.226 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:43.027 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:12.235 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:57.428 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:26.637 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:57.482 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:26.691 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:57.484 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:26.692 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:57.487 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:26.696 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:57.525 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:26.733 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:57.532 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:26.741 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:57.541 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:26.708 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:57.545 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:26.713 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:57.578 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:26.745 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:50:58.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (23049 ms) 2024-05-15T12:50:58.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-05-15T12:50:58.585 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25076 ms) 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4352 ms) 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5362 ms) 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (5590 ms) 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (5630 ms) 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-05-15T12:50:58.586 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4894 ms) 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39899 ms) 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (20014 ms) 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5935 ms) 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (5048 ms) 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-05-15T12:50:58.587 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26284 ms) 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (17138 ms) 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6395 ms) 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5389 ms) 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: waiting for scrubs... 2024-05-15T12:50:58.588 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: done waiting 2024-05-15T12:50:58.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26487 ms) 2024-05-15T12:50:58.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-05-15T12:50:58.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12619 ms) 2024-05-15T12:50:58.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-05-15T12:50:58.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11091 ms) 2024-05-15T12:50:58.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-05-15T12:50:58.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:50:58.589 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11279 ms) 2024-05-15T12:51:02.370 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:31.578 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:02.374 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:31.582 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:02.389 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:31.597 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:02.418 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:31.585 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:02.420 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:31.587 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:02.430 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:31.639 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:02.433 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:31.601 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:02.435 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:31.643 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:02.449 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:31.658 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:12.406 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:41.614 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:12.459 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:41.668 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:12.485 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:41.694 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:12.508 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:41.675 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:12.510 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:41.677 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:12.530 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:41.738 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:12.533 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:41.701 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:12.534 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:41.742 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:12.559 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:41.768 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:22.425 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:51.634 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:22.474 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:51.641 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:22.476 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:51.643 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:22.483 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:51.692 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:22.502 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:51.669 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:22.510 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:51.718 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:22.574 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:51.783 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:22.577 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:51.786 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:22.613 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:51.822 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:27.690 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:56.899 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:27.697 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:56.906 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:27.725 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:19:56.933 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:27.756 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:56.923 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:27.768 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:56.977 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:27.781 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:56.988 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:27.807 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:19:57.015 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:27.855 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:57.022 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:27.881 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:19:57.048 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:42.619 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:11.828 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:42.626 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:11.835 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:42.662 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:11.871 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:42.683 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:11.850 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:42.687 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:11.854 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:42.725 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:11.892 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:42.730 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:11.935 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:42.731 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:11.940 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:42.764 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:11.972 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:47.544 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:16.752 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:47.547 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:16.756 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:47.560 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:16.769 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:47.593 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:16.760 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:47.598 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:16.765 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:47.598 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:16.807 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:47.601 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:16.810 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:47.612 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:16.779 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:51:47.613 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:16.822 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:02.548 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:31.755 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:02.596 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:31.763 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:02.597 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:31.765 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:02.600 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:31.808 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:02.626 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:31.793 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:02.642 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:31.851 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:02.667 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:31.876 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:02.671 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:31.880 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:02.712 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:31.921 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:03.078 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:32.245 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:03.079 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:32.246 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:03.086 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:32.254 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:03.187 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:32.396 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:03.188 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:32.397 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:03.197 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:32.405 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:03.247 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:32.456 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:03.248 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:32.457 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:03.257 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:32.466 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:06.656 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T12:52:06.660+0000 26292640 -1 snap_mapper.add_oid found existing snaps mapped on 104:ce1a3b0a:test-rados-api-smithi049-40250-97::foo:21, removing 2024-05-15T12:52:06.663 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T12:52:06.641+0000 2228a640 -1 snap_mapper.add_oid found existing snaps mapped on 104:ce1a3b0a:test-rados-api-smithi049-40250-97::foo:21, removing 2024-05-15T12:52:17.636 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:46.844 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:17.639 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:46.848 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:17.678 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:46.887 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:17.691 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:46.858 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:17.694 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:46.862 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:17.695 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:46.903 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:17.699 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:46.908 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:17.725 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:46.891 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:17.731 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:46.940 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:18.537 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:47.704 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:18.546 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:47.713 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:18.560 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:47.728 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:18.681 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:47.889 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:18.682 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:47.890 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:18.697 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:47.905 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:18.792 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:48.000 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:18.793 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:48.001 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:18.808 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:48.017 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:22.649 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:51.857 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:22.653 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:51.862 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:22.683 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:20:51.892 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:22.700 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:51.867 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:22.705 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:51.873 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:22.719 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:51.928 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:22.721 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:20:51.888 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:22.725 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:51.933 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:22.739 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:20:51.948 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:32.778 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:01.986 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:32.815 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:02.024 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:32.826 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:01.993 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:32.828 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:01.995 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:32.837 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:02.045 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:32.838 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:02.046 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:32.846 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:02.054 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:32.854 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:02.021 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:32.865 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:02.074 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:42.957 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:12.165 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:42.963 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:12.172 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:42.989 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:12.197 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:43.013 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:12.180 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:43.015 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:12.182 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:43.033 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:12.242 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:43.036 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:12.244 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:43.040 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:12.207 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:43.065 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:12.274 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:47.837 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:17.045 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:47.839 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:17.048 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:47.854 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:17.063 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:47.886 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:17.053 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:47.887 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:17.054 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:47.902 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:17.069 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:47.909 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:17.117 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:47.911 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:17.120 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:52:47.926 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:17.135 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:02.906 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:32.113 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:02.915 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:32.121 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:02.947 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:32.156 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:02.951 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:32.118 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:02.953 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:32.120 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:02.955 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:32.164 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:02.959 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:32.168 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:02.988 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:32.156 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:02.996 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:32.204 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:03.746 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:32.913 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:03.747 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:32.914 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:03.757 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:32.924 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:03.883 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:33.092 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:03.884 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:33.093 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:03.895 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:33.103 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:03.936 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:33.144 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:03.936 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:33.145 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:03.947 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:33.156 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:18.097 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:47.264 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:18.098 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:47.265 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:18.102 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:47.311 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:18.104 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:47.313 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:18.135 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:47.303 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:18.150 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:47.358 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:18.173 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:47.381 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:18.183 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:47.391 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:18.233 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:47.441 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:22.980 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:52.188 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:22.981 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:52.190 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:22.996 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:21:52.205 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:23.027 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:52.194 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:23.028 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:52.196 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:23.032 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:52.241 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:23.033 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:52.242 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:23.042 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:21:52.210 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:23.048 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:21:52.256 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:33.049 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:02.258 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:33.055 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:02.264 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:33.100 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:02.308 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:33.104 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:02.272 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:33.106 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:02.274 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:33.129 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:02.338 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:33.130 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:02.339 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:33.142 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:02.309 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:33.158 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:02.367 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:43.272 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:12.438 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:43.273 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:12.440 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:43.283 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:12.492 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:43.286 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:12.494 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:43.303 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:12.470 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:43.334 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:12.542 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:43.353 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:12.561 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:43.357 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:12.566 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:43.385 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:12.593 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:48.153 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:17.362 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:48.155 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:17.364 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:48.179 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:17.388 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:48.219 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:17.386 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:48.221 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:17.388 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:48.242 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:17.409 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:48.244 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:17.453 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:48.246 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:17.455 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:53:48.270 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:17.479 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.173 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:32.381 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.175 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:32.384 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.202 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:32.410 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.224 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:32.432 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.226 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:32.394 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.229 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:32.396 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.229 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:32.438 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.253 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:32.420 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.255 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:32.464 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.868 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:33.035 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.869 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:33.036 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.878 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:33.045 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.911 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:33.120 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.912 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:33.121 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.921 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:33.129 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.952 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:33.160 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.953 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:33.161 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:03.963 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:33.171 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:23.378 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:52.587 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:23.381 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:52.590 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:23.426 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:52.634 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:23.446 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:52.655 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:23.447 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:52.656 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:23.448 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:52.615 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:23.450 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:52.617 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:23.495 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:52.662 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:23.496 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:52.705 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:27.281 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:56.489 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:27.282 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:56.491 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:27.310 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:56.519 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:27.338 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:56.505 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:27.339 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:56.506 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:27.344 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:56.552 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:27.346 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:56.555 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:27.360 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:56.527 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:27.374 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:56.583 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:28.408 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:57.617 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:28.412 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:57.620 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:28.431 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:22:57.639 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:28.483 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:57.692 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:28.489 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:57.656 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:28.490 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:57.657 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:28.498 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:57.707 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:28.500 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:22:57.667 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:28.512 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:22:57.721 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:38.411 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:07.619 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:38.413 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:07.622 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:38.439 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:07.648 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:38.467 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:07.634 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:38.468 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:07.677 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:38.473 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:07.681 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:38.476 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:07.643 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:38.498 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:07.707 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:38.503 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:07.671 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:48.606 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:17.815 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:48.613 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:17.821 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:48.647 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:17.856 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:48.666 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:17.833 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:48.668 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:17.835 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:48.674 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:17.883 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:48.677 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:17.885 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:48.696 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:17.863 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:48.714 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:17.923 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:53.463 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:22.671 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:53.467 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:22.676 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:53.483 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:22.691 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:53.512 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:22.679 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:53.514 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:22.681 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:53.517 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:22.726 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:53.520 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:22.729 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:53.530 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:22.698 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:54:53.536 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:22.745 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:08.520 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:37.729 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:08.524 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:37.733 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:08.559 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:37.768 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:08.580 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:37.748 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:08.581 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:37.790 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:08.582 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:37.749 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:08.585 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:37.794 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:08.615 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:37.782 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:08.619 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:37.828 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:09.457 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:38.624 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:09.458 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:38.625 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:09.469 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:38.636 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:09.514 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:38.723 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:09.516 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:38.724 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:09.526 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:38.735 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:09.575 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:38.784 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:09.576 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:38.785 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:09.586 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:38.795 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:10.943 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-05-15T12:55:10.943 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:55:10.943 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (10964 ms) 2024-05-15T12:55:10.943 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-05-15T12:55:10.943 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:55:10.943 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (9019 ms) 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (7064 ms) 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26962 ms) 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15042 ms) 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (13059 ms) 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: 00000010 69 73 |is| 2024-05-15T12:55:10.944 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: 00000012 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (14186 ms) 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: require_osd_release = squid 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7818 ms) 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (951865 ms total) 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1704 ms) 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10815 ms) 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (20273 ms) 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-05-15T12:55:10.945 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (73 ms) 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (32865 ms total) 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8987 ms) 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (10000 ms) 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: waiting for scrub... 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: done waiting 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (29053 ms) 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-05-15T12:55:10.946 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (12391 ms) 2024-05-15T12:55:10.947 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-05-15T12:55:10.947 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (10755 ms) 2024-05-15T12:55:10.947 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-05-15T12:55:10.947 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (10575 ms) 2024-05-15T12:55:10.947 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-05-15T12:55:10.947 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12437 ms) 2024-05-15T12:55:10.947 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-05-15T12:55:23.794 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:53.003 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:23.797 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:53.006 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:23.827 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:53.036 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:23.847 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:53.014 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:23.848 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:53.015 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:23.860 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:53.068 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:23.861 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:53.070 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:23.874 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:53.041 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:23.892 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:53.101 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:28.669 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:57.878 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:28.671 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:57.880 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:28.685 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:23:57.893 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:28.717 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:57.885 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:28.719 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:57.886 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:28.722 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:57.930 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:28.723 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:57.932 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:28.732 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:23:57.899 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:28.738 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:23:57.947 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:38.876 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:08.085 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:38.880 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:08.089 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:38.922 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:08.131 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:38.942 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:08.110 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:38.954 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:08.162 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:38.958 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:08.125 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:38.959 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:08.168 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:38.996 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:08.205 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:38.998 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:08.166 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:48.909 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:18.117 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:48.911 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:18.120 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:48.954 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:18.163 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:48.958 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:18.125 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:48.959 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:18.127 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:48.978 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:18.186 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:48.981 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:18.149 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:48.989 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:18.198 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:49.012 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:18.221 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:53.889 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:23.098 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:53.890 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:23.099 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:53.914 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:23.123 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:53.943 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:23.111 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:53.946 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:23.114 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:53.964 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:23.173 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:53.966 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:23.174 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:53.966 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:23.133 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:55:53.990 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:23.199 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:08.851 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:38.060 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:08.856 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:38.065 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:08.909 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:38.076 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:08.910 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:38.078 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:08.925 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:38.134 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:08.928 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:38.137 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:08.947 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:38.114 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:08.955 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:38.164 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:08.981 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:38.188 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:09.730 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:38.898 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:09.732 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:38.899 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:09.741 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:38.908 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:09.791 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:38.999 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:09.792 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:39.001 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:09.801 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:39.009 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:09.817 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:39.026 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:09.818 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:39.027 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:09.827 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:39.036 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:23.913 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:53.122 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:23.915 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:53.124 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:23.950 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:53.159 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:23.966 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:53.134 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:23.967 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:53.135 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:23.975 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:53.184 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:23.977 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:53.186 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:24.000 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:53.167 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:24.022 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:53.231 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:29.081 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:58.290 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:29.084 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:58.293 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:29.100 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:24:58.309 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:29.137 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:58.346 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:29.138 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:58.347 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:29.148 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:58.315 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:29.151 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:58.319 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:29.154 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:24:58.363 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:29.169 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:24:58.336 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:33.698 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:02.906 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:33.700 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:02.909 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:33.712 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:02.879 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:33.716 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:02.883 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:33.718 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:02.927 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:33.781 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:02.990 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:33.787 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:02.996 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:33.809 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:03.018 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:33.814 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:02.981 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:38.966 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:08.175 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:38.970 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:08.178 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:39.005 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:08.214 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:39.018 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:08.186 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:39.020 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:08.187 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:39.025 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:08.234 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:39.031 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:08.239 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:39.034 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:08.202 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:39.049 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:08.258 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:49.102 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:18.311 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:49.104 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:18.313 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:49.121 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:18.330 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:49.152 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:18.320 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:49.153 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:18.321 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:49.158 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:18.366 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:49.160 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:18.369 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:49.167 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:18.334 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:49.174 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:18.383 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:54.312 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:23.520 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:54.337 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:23.546 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:54.375 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:23.542 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:54.389 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:23.597 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:54.389 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:23.598 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:54.391 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:23.558 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:54.410 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:23.618 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:54.417 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:23.584 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:56:54.459 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:23.668 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:09.270 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:38.479 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:09.278 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:38.487 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:09.282 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:38.449 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:09.283 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:38.492 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:09.285 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:38.452 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:09.286 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:38.495 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:09.371 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:38.538 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:09.393 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:38.602 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:09.399 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:38.608 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:14.200 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:43.408 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:14.202 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:43.411 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:14.215 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:43.424 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:14.250 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:43.418 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:14.252 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:43.419 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:14.253 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:43.461 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:14.254 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:43.463 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:14.266 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:43.433 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:14.267 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:43.476 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:24.177 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:53.386 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:24.233 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:53.442 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:24.257 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:25:53.466 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:24.283 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:53.450 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:24.284 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:53.493 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:24.284 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:53.452 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:24.286 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:53.495 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:24.310 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:25:53.477 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:24.312 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:25:53.521 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:34.410 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:03.619 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:34.418 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:03.627 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:34.467 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:03.634 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:34.471 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:03.638 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:34.480 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:03.689 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:34.485 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:03.693 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:34.487 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:03.655 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:34.498 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:03.705 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:34.504 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:03.713 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:39.463 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:08.671 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:39.469 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:08.678 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:39.505 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:08.714 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:39.530 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:08.739 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:39.532 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:08.700 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:39.538 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:08.746 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:39.539 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:08.706 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:39.558 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:08.725 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:39.570 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:08.779 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:54.320 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:23.527 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:54.374 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:23.582 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:54.419 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:23.628 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:54.421 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:23.588 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:54.424 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:23.592 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:54.446 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:23.655 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:54.449 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:23.657 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:54.452 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:23.620 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:54.478 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:23.687 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:55.123 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:24.290 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:55.124 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:24.291 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:55.132 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:24.299 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:55.253 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:24.462 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:55.254 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:24.463 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:55.263 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:24.472 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:55.294 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:24.503 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:55.295 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:24.504 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:57:55.304 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:24.513 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:09.571 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:38.779 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:09.644 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:38.812 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:09.645 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:38.854 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:09.652 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:38.820 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:09.685 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:38.894 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:09.701 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:38.868 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:09.717 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:38.926 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:09.721 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:38.930 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:09.763 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:38.972 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:14.671 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:43.880 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:14.674 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:43.883 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:14.700 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:43.909 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:14.731 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:43.899 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:14.735 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:43.903 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:14.745 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:43.954 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:14.749 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:43.957 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:14.756 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:43.923 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:14.779 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:43.987 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:24.671 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:53.880 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:24.674 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:53.883 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:24.706 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:53.914 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:24.708 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:53.917 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:24.736 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:53.903 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:24.742 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:53.910 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:24.750 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:53.959 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:24.778 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:53.986 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:24.785 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:53.952 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:26.786 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10015 ms) 2024-05-15T12:58:26.786 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-05-15T12:58:26.786 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (15699 ms) 2024-05-15T12:58:26.786 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-05-15T12:58:26.786 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (11280 ms) 2024-05-15T12:58:26.786 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-05-15T12:58:26.786 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (22456 ms) 2024-05-15T12:58:26.786 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-05-15T12:58:26.786 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9948 ms) 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9986 ms) 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: hmm, no HitSet yet 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: ok, hit_set contains 221:602f83fe:::foo:head 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9974 ms) 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,0 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: first is 1715777799 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,0 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,0 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,1715777801,1715777802,0 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,1715777801,1715777802,0 2024-05-15T12:58:26.787 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,1715777801,1715777802,0 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,1715777801,1715777802,1715777804,1715777805,0 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,1715777801,1715777802,1715777804,1715777805,0 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,1715777801,1715777802,1715777804,1715777805,0 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,1715777801,1715777802,1715777804,1715777805,1715777807,1715777808,0 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,1715777801,1715777802,1715777804,1715777805,1715777807,1715777808,0 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777799,1715777801,1715777802,1715777804,1715777805,1715777807,1715777808,0 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: got ls 1715777802,1715777804,1715777805,1715777807,1715777808,1715777811,1715777812,0 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: first now 1715777802, trimmed 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (25746 ms) 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: foo0 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-05-15T12:58:26.788 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (18908 ms) 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (21854 ms) 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (24668 ms) 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6582 ms) 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (5660 ms) 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (5431 ms) 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4616 ms) 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (297021 ms total) 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: 2024-05-15T12:58:26.789 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-05-15T12:58:26.790 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1438678 ms total) 2024-05-15T12:58:26.790 INFO:tasks.workunit.client.0.smithi049.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-05-15T12:58:26.791 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:58:26.791 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39939 2024-05-15T12:58:26.791 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 39939 2024-05-15T12:58:26.791 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40159 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40159 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40640 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40640 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40405 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40405 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40274 2024-05-15T12:58:26.792 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40274 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40018 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40018 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40325 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40325 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40870 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40870 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40891 2024-05-15T12:58:26.793 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40891 2024-05-15T12:58:29.702 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:58.910 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:29.710 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:58.919 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:29.732 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:26:58.941 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:29.770 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:58.937 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:29.772 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:58.940 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:29.773 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:58.982 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:29.775 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:58.983 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:29.792 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:26:58.960 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:29.796 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:26:59.005 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:34.708 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:03.917 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:34.711 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:03.920 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:34.723 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:03.932 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:34.761 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:03.929 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:34.762 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:03.930 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:34.774 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:03.941 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:34.785 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:03.994 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:34.787 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:03.996 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:34.798 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:04.007 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:44.684 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:13.893 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:44.732 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:13.900 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:44.734 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:13.901 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:44.736 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:13.945 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:44.749 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:13.916 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:44.751 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:13.959 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:44.789 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:13.998 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:44.790 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:13.999 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:44.803 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:14.012 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:49.755 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:18.963 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:49.757 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:18.966 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:49.768 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:18.977 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:49.807 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:19.016 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:49.808 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:19.017 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:49.811 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:18.978 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:49.812 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:18.979 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:49.820 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:19.029 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:49.825 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:18.992 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:54.765 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:23.973 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:54.766 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:23.975 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:54.777 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:23.986 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:54.813 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:23.980 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:54.814 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:23.981 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:54.816 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:24.025 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:54.819 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:24.028 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:54.824 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:23.992 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:58:54.829 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:24.038 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:04.788 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:33.996 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:04.792 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:34.001 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:04.807 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:34.016 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:04.840 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:34.007 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:04.841 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:34.009 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:04.845 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:34.054 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:04.846 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:34.055 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:04.853 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:34.021 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:04.864 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:34.073 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:05.854 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:35.063 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:05.865 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:35.032 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:05.866 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:35.075 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:05.902 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:35.111 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:05.910 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:35.077 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:05.915 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:35.124 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:05.927 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:35.095 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:05.931 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:35.140 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:05.933 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:35.142 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:14.818 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:44.024 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:14.818 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:44.026 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:14.828 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:44.037 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:14.864 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:44.031 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:14.865 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:44.032 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:14.868 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:44.077 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:14.869 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:44.078 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:14.875 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:44.043 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:14.881 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:44.090 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:24.838 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:54.047 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:24.840 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:54.049 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:24.859 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:54.068 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:24.892 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:54.059 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:24.895 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:54.062 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:24.897 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:54.105 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:24.898 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:54.107 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:24.907 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:54.075 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:24.912 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:54.121 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:29.879 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:59.088 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:29.880 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:59.089 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:29.893 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:27:59.102 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:29.926 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:59.094 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:29.927 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:59.095 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:29.934 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:59.143 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:29.935 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:59.144 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:29.941 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:27:59.107 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:29.948 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:27:59.157 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:34.885 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:04.094 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:34.917 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:04.126 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:34.927 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:04.136 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:34.936 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:04.104 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:34.938 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:04.105 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:34.939 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:04.148 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:34.940 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:04.149 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:34.949 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:04.117 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:34.950 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:04.159 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:44.910 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:14.119 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:44.913 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:14.121 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:44.956 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:14.165 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:44.962 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:14.130 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:44.964 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:14.132 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:44.965 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:14.173 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:44.967 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:14.176 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:44.978 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:14.146 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:44.985 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:14.194 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:54.978 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:24.186 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:54.981 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:24.190 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:54.996 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:24.205 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:55.028 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:24.196 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:55.030 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:24.197 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:55.031 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:24.240 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:55.032 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:24.241 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:55.043 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:24.210 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T12:59:55.046 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:24.255 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:05.000 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:34.208 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:05.002 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:34.211 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:05.020 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:34.229 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:05.053 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:34.220 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:05.054 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:34.222 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:05.058 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:34.267 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:05.060 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:34.269 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:05.067 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:34.235 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:05.073 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:34.282 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.028 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:39.237 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.032 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:39.242 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.044 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:39.253 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.081 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:39.248 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.082 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:39.250 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.086 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:39.295 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.087 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:39.296 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.095 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:39.262 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:10.101 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:39.310 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:15.013 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:44.222 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:15.039 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:44.248 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:15.053 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:44.262 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:15.088 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:44.256 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:15.089 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:44.257 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:15.094 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:44.303 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:15.098 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:44.304 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:15.099 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:44.267 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:15.105 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:44.314 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:25.060 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:54.269 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:25.064 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:54.273 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:25.077 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:28:54.286 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:25.114 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:54.281 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:25.115 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:54.324 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:25.118 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:54.286 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:25.122 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:54.331 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:25.131 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:28:54.299 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:25.137 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:28:54.346 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:35.087 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:04.296 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:35.089 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:04.298 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:35.102 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:04.311 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:35.141 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:04.308 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:35.142 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:04.310 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:35.154 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:04.322 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:35.159 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:04.368 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:35.160 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:04.369 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:35.173 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:04.382 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:45.153 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:14.362 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:45.155 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:14.365 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:45.171 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:14.380 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:45.204 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:14.371 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:45.205 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:14.373 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:45.217 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:14.385 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:45.220 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:14.429 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:45.225 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:14.434 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:45.238 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:14.447 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:55.141 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:24.350 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:55.143 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:24.352 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:55.155 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:24.364 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:55.191 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:24.359 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:55.193 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:24.360 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:55.194 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:24.403 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:55.195 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:24.405 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:55.205 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:24.372 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:00:55.208 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:24.417 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:00.141 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:29.350 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:00.194 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:29.403 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:00.207 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:29.416 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:00.250 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:29.417 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:00.250 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:29.459 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:00.251 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:29.419 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:00.252 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:29.461 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:00.264 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:29.432 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:00.265 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:29.474 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:05.199 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:34.408 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:05.201 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:34.410 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:05.211 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:34.420 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:05.249 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:34.416 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:05.250 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:34.418 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:05.253 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:34.462 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:05.254 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:34.463 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:05.260 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:34.428 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:05.264 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:34.473 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:15.221 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:44.430 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:15.225 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:44.434 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:15.237 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:44.446 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:15.275 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:44.484 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:15.275 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:44.443 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:15.279 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:44.488 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:15.279 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:44.447 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:15.293 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:44.460 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:15.293 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:44.501 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:25.242 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:54.450 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:25.244 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:54.453 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:25.256 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:29:54.465 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:25.291 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:54.458 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:25.292 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:54.460 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:25.293 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:54.502 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:25.295 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:54.504 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:25.304 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:29:54.472 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:25.307 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:29:54.516 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:35.266 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:04.474 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:35.268 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:04.478 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:35.282 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:04.491 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:35.315 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:04.483 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:35.316 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:04.484 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:35.318 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:04.527 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:35.320 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:04.529 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:35.329 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:04.497 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:35.333 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:04.542 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:45.287 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:14.496 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:45.291 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:14.500 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:45.303 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:14.512 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:45.339 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:14.548 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:45.340 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:14.508 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:45.343 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:14.553 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:45.344 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:14.512 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:45.358 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:14.526 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:01:45.360 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:14.568 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:00.324 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:29.533 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:00.329 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:29.538 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:00.356 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:29.565 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:00.375 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:29.543 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:00.377 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:29.544 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:00.387 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:29.596 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:00.388 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:29.598 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:00.394 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:29.561 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:00.406 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:29.615 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:05.344 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:34.553 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:05.350 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:34.559 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:05.366 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:34.575 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:05.393 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:34.560 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:05.397 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:34.564 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:05.408 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:34.575 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:05.421 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:34.630 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:05.425 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:34.634 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:05.437 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:34.646 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:20.373 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:49.582 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:20.377 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:49.586 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:20.402 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:49.611 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:20.428 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:49.596 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:20.429 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:49.597 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:20.434 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:49.644 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:20.437 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:49.646 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:20.446 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:49.614 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:20.454 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:49.663 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:25.386 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:54.595 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:25.390 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:54.599 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:25.406 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:30:54.615 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:25.436 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:54.603 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:25.439 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:54.607 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:25.451 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:30:54.618 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:25.470 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:54.677 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:25.471 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:54.680 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:25.483 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:30:54.692 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:40.420 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:09.629 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:40.422 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:09.632 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:40.439 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:09.648 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:40.471 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:09.638 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:40.472 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:09.639 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:40.477 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:09.686 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:40.478 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:09.688 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:40.489 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:09.657 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:40.503 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:09.712 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:45.441 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:14.650 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:45.444 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:14.653 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:45.454 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:14.663 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:45.485 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:14.652 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:45.489 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:14.657 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:45.491 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:14.701 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:45.495 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:14.704 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:45.500 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:14.668 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:45.507 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:14.716 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:49.737 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:18.946 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:49.741 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:18.950 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:49.747 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:18.956 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:49.748 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:18.916 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:49.749 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:18.959 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:49.750 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:18.918 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:49.763 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:18.972 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:49.767 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:18.935 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:02:49.772 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:18.981 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:00.479 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:29.688 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:00.483 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:29.693 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:00.498 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:29.707 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:00.548 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:29.716 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:00.552 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:29.720 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:00.553 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:29.762 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:00.555 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:29.764 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:00.565 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:29.733 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:00.571 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:29.780 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:05.445 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:34.654 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:05.500 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:34.709 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:05.521 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:34.730 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:05.549 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:34.716 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:05.553 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:34.720 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:05.557 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:34.766 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:05.560 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:34.769 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:05.564 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:34.732 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:05.572 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:34.781 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:20.534 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:49.743 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:20.537 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:49.747 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:20.571 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:49.780 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:20.590 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:49.800 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:20.592 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:49.801 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:20.597 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:49.765 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:20.598 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:49.766 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:20.614 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:49.823 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:20.615 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:49.782 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:25.553 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:54.762 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:25.555 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:54.764 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:25.573 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:31:54.782 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:25.606 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:54.773 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:25.608 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:54.817 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:25.609 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:54.776 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:25.613 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:54.822 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:25.620 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:31:54.788 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:25.625 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:31:54.834 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:40.581 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:09.790 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:40.611 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:09.820 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:40.628 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:09.838 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:40.639 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:09.807 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:40.640 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:09.808 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:40.656 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:09.865 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:40.657 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:09.824 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:40.658 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:09.867 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:40.674 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:09.883 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:45.590 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:14.799 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:45.593 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:14.802 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:45.607 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:14.816 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:45.639 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:14.806 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:45.640 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:14.808 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:45.651 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:14.818 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:45.659 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:14.869 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:45.661 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:14.870 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:03:45.677 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:14.886 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:05.692 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:34.901 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:05.693 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:34.902 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:05.715 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:34.924 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:05.741 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:34.909 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:05.743 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:34.911 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:05.748 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:34.957 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:05.751 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:34.960 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:05.762 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:34.930 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:05.772 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:34.981 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:06.338 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:35.505 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:06.339 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:35.506 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:06.346 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:35.514 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:06.356 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:35.565 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:06.357 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:35.566 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:06.369 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:35.578 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:06.393 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:35.602 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:06.394 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:35.603 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:06.402 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:35.611 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:25.738 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:54.947 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:25.739 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:54.948 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:25.758 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:54.967 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:25.783 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:54.951 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:25.786 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:54.954 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:25.791 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:55.000 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:25.793 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:55.002 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:25.807 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:54.975 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:25.813 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:55.022 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:26.310 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:55.518 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:26.310 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:55.478 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:26.311 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:55.520 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:26.311 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:55.479 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:26.319 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:32:55.528 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:26.319 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:32:55.487 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:26.355 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:55.564 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:26.356 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:55.566 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:26.364 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:32:55.574 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:45.801 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:15.010 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:45.802 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:15.011 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:45.821 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:15.030 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:45.855 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:15.023 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:45.857 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:15.025 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:45.858 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:15.068 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:45.860 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:15.069 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:45.875 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:15.043 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:45.879 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:15.089 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:46.395 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:15.563 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:46.396 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:15.564 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:46.411 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:15.579 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:46.508 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:15.717 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:46.520 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:15.729 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:46.532 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:15.741 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:46.555 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:15.764 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:46.560 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:15.769 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:46.651 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:15.860 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:04:53.087 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: Running main() from gmock_main.cc 2024-05-15T13:04:53.087 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-05-15T13:04:53.087 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [----------] Global test environment set-up. 2024-05-15T13:04:53.087 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-05-15T13:04:53.087 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-05-15T13:04:53.087 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (10485 ms) 2024-05-15T13:04:53.087 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-05-15T13:04:53.087 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (1814146 ms) 2024-05-15T13:04:53.088 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (1824631 ms total) 2024-05-15T13:04:53.088 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: 2024-05-15T13:04:53.088 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [----------] Global test environment tear-down 2024-05-15T13:04:53.088 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (1824631 ms total) 2024-05-15T13:04:53.088 INFO:tasks.workunit.client.0.smithi049.stdout: ec_io: [ PASSED ] 2 tests. 2024-05-15T13:04:53.089 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:04:53.089 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40039 2024-05-15T13:04:53.089 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40039 2024-05-15T13:04:53.089 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:04:53.089 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40656 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40656 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40536 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40536 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39914 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 39914 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39973 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 39973 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:04:53.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40589 2024-05-15T13:04:53.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40589 2024-05-15T13:04:53.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:04:53.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39905 2024-05-15T13:04:53.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 39905 2024-05-15T13:05:00.849 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:30.058 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:00.852 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:30.061 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:00.886 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:30.095 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:00.906 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:30.073 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:00.911 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:30.078 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:00.920 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:30.129 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:00.926 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:30.136 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:00.947 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:30.115 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:00.973 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:30.182 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:06.194 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:35.403 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:06.209 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:35.418 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:06.239 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:35.448 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:06.247 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:35.456 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:06.262 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:35.430 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:06.266 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:35.434 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:06.301 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:35.469 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:06.303 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:35.513 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:06.340 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:35.549 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:16.180 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:45.390 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:16.184 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:45.393 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:16.230 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:33:45.439 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:16.235 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:45.403 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:16.237 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:45.405 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:16.258 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:45.467 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:16.262 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:45.471 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:16.270 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:33:45.438 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:16.300 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:33:45.510 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:24.813 INFO:tasks.workunit.client.0.smithi049.stdout: misc: Running main() from gmock_main.cc 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [----------] Global test environment set-up. 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.Version (6768 ms) 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (5431 ms) 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.LongName (1811283 ms) 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3429 ms) 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (4629 ms) 2024-05-15T13:05:24.814 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3438 ms) 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.Exec (3717 ms) 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3071 ms) 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (4254 ms) 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3667 ms) 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2175 ms) 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (4444 ms) 2024-05-15T13:05:24.815 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [----------] 12 tests from NeoRadosMisc (1856306 ms total) 2024-05-15T13:05:24.816 INFO:tasks.workunit.client.0.smithi049.stdout: misc: 2024-05-15T13:05:24.816 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [----------] Global test environment tear-down 2024-05-15T13:05:24.816 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [==========] 12 tests from 1 test suite ran. (1856307 ms total) 2024-05-15T13:05:24.816 INFO:tasks.workunit.client.0.smithi049.stdout: misc: [ PASSED ] 12 tests. 2024-05-15T13:05:31.314 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:00.482 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:31.317 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:00.485 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:31.321 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:00.530 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:31.340 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:00.550 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:31.366 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:00.534 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:31.389 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:00.598 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:31.396 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:00.606 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:31.403 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:00.612 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:31.472 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:00.681 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.426 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:12.636 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.434 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:12.643 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.474 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:12.684 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.491 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:12.700 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.505 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:12.673 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.507 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:12.674 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.531 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:12.740 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.542 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:12.710 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:43.583 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:12.792 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:46.148 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:15.357 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:46.201 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:15.369 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:46.204 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:15.371 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:46.216 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:15.383 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:46.226 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:15.435 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:46.255 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:15.464 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:46.299 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:15.508 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:46.302 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:15.512 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:46.324 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:15.533 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:51.339 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:20.548 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:51.347 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:20.515 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:51.350 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:20.518 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:51.351 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:20.560 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:51.362 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:20.569 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:51.363 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:20.572 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:51.437 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:20.605 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:51.467 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:20.677 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:05:51.472 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:20.682 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:01.490 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:30.699 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:01.500 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:30.709 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:01.538 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:30.747 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:01.567 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:30.735 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:01.575 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:30.784 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:01.582 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:30.750 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:01.585 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:30.794 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:01.615 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:30.783 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:01.623 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:30.833 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:11.308 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:40.517 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:11.310 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:40.520 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:11.337 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:40.547 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:11.357 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:40.525 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:11.359 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:40.527 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:11.384 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:40.552 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:11.384 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:40.594 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:11.386 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:40.596 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:11.411 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:40.620 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:16.313 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:45.523 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:16.315 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:45.525 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:16.331 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:34:45.540 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:16.360 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:45.528 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:16.361 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:45.529 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:16.366 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:45.575 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:16.368 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:45.577 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:16.375 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:34:45.543 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:16.383 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:34:45.592 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:16.962 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: Running main() from gmock_main.cc 2024-05-15T13:06:16.962 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [----------] Global test environment set-up. 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.TooBig (7318 ms) 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (1811138 ms) 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3342 ms) 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (4299 ms) 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3589 ms) 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3811 ms) 2024-05-15T13:06:16.963 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3184 ms) 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3569 ms) 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4058 ms) 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (10490 ms) 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3344 ms) 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3212 ms) 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2935 ms) 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3052 ms) 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-05-15T13:06:16.964 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.Flush (3104 ms) 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2856 ms) 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (4338 ms) 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3274 ms) 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3488 ms) 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (4098 ms) 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3097 ms) 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-05-15T13:06:16.965 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3066 ms) 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3253 ms) 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3460 ms) 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3663 ms) 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3966 ms) 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [----------] 26 tests from LibRadosAio (1909005 ms total) 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: 2024-05-15T13:06:16.966 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-05-15T13:06:31.415 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:00.624 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:31.418 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:00.628 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:31.465 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:00.633 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:31.468 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:00.677 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:31.468 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:00.636 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:31.493 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:00.702 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:31.494 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:00.704 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:31.502 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:00.670 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:31.577 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:00.786 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:32.283 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:01.451 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:32.284 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:01.452 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:32.296 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:01.464 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:32.330 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:01.539 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:32.331 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:01.540 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:32.343 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:01.552 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:32.394 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:01.603 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:32.395 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:01.605 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:32.409 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:01.618 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:43.650 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleW api_aio_pp: Running main() from gmock_main.cc 2024-05-15T13:06:43.650 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-05-15T13:06:43.650 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-05-15T13:06:43.650 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-05-15T13:06:43.650 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (7176 ms) 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (1839345 ms) 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (7117 ms) 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (5656 ms) 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3282 ms) 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2715 ms) 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3141 ms) 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-05-15T13:06:43.651 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3056 ms) 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3864 ms) 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3512 ms) 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (4292 ms) 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3353 ms) 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3020 ms) 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3083 ms) 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3142 ms) 2024-05-15T13:06:43.652 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3296 ms) 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3669 ms) 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3558 ms) 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3803 ms) 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3418 ms) 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3366 ms) 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2657 ms) 2024-05-15T13:06:43.653 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-05-15T13:06:43.654 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3340 ms) 2024-05-15T13:06:43.654 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-05-15T13:06:43.654 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3559 ms) 2024-05-15T13:06:43.654 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-05-15T13:06:43.654 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3279 ms) 2024-05-15T13:06:43.654 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-05-15T13:06:43.654 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3264 ms) 2024-05-15T13:06:43.654 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-05-15T13:06:46.476 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:15.686 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:46.478 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:15.688 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:46.507 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:15.717 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:46.531 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:15.698 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:46.531 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:15.741 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:46.532 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:15.700 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:46.533 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:15.742 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:46.561 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:15.729 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:06:46.564 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:15.773 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:01.594 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:30.803 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:01.611 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:30.821 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:01.646 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:30.814 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:01.651 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:30.819 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:01.658 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:30.868 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:01.667 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:30.877 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:01.685 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:30.894 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:01.723 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:30.891 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:01.736 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:30.946 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:16.607 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:45.816 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:16.626 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:45.835 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:16.653 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:45.821 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:16.659 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:45.827 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:16.662 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:45.871 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:16.664 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:45.873 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:16.691 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:45.900 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:16.695 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:45.863 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:16.703 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:45.913 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:17.253 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:46.421 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:17.260 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:46.428 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:17.338 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:35:46.505 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:17.383 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:46.592 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:17.384 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:46.593 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:17.396 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:35:46.605 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:17.512 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:46.721 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:17.513 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:46.723 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:17.540 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:35:46.749 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:31.987 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:01.196 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:31.995 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:01.204 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:32.056 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:01.223 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:32.056 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:01.265 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:32.058 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:01.226 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:32.080 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:01.289 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:32.082 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:01.292 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:32.104 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:01.272 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:32.144 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:01.354 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:36.803 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:06.012 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:36.807 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:06.016 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:36.825 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:06.034 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:36.855 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:06.023 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:36.856 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:06.024 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:36.870 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:06.038 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:36.874 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:06.084 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:36.876 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:06.086 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:36.897 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:06.106 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:38.722 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:07.932 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:38.777 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:07.986 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:38.794 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:08.004 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:38.829 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:07.997 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:38.832 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:08.000 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:38.836 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:08.045 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:38.837 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:08.046 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:38.848 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:08.016 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:38.853 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:08.063 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:46.903 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:16.111 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:46.905 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:16.114 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:46.928 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:16.137 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:46.951 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:16.119 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:46.952 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:16.120 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:46.957 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:16.167 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:46.959 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:16.169 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:46.974 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:16.142 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:46.982 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:16.191 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:56.969 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:26.178 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:56.971 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:26.181 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:56.997 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:26.207 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:57.020 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:26.187 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:57.021 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:26.189 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:57.023 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:26.233 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:57.025 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:26.235 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:57.049 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:26.217 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:07:57.056 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:26.265 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:07.093 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:36.302 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:07.096 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:36.305 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:07.140 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:36.350 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:07.146 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:36.314 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:07.150 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:36.318 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:07.152 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:36.362 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:07.157 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:36.366 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:07.178 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:36.346 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:07.188 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:36.398 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:17.121 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:46.330 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:17.126 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:46.333 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:17.158 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:46.367 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:17.176 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:46.344 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:17.177 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:46.345 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:17.180 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:46.390 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:17.181 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:46.391 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:17.201 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:46.369 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:17.207 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:46.416 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:18.088 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:47.256 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:18.089 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:47.257 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:18.098 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:36:47.266 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:18.138 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:47.348 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:18.139 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:47.349 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:18.150 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:36:47.359 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:18.184 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:47.393 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:18.185 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:47.394 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:18.198 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:36:47.407 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:32.343 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:01.553 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:32.348 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:01.558 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:32.386 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:01.596 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:32.409 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:01.577 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:32.411 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:01.579 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:32.418 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:01.628 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:32.420 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:01.630 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:32.452 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:01.620 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:32.454 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:01.664 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:37.199 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:06.408 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:37.201 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:06.410 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:37.226 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:06.436 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:37.244 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:06.412 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:37.246 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:06.414 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:37.260 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:06.469 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:37.260 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:06.428 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:37.261 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:06.470 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:37.275 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:06.485 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:47.398 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:16.607 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:47.402 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:16.612 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:47.430 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:16.639 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:47.450 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:16.618 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:47.451 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:16.619 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:47.476 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:16.644 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:47.480 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:16.689 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:47.481 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:16.690 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:47.517 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:16.727 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:49.571 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosArite 2024-05-15T13:08:49.571 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (39182 ms) 2024-05-15T13:08:49.571 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-05-15T13:08:49.571 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (9367 ms) 2024-05-15T13:08:49.571 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-05-15T13:08:49.571 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7207 ms) 2024-05-15T13:08:49.571 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7664 ms) 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7775 ms) 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8223 ms) 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6600 ms) 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7700 ms) 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6806 ms) 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7704 ms) 2024-05-15T13:08:49.572 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6939 ms) 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7546 ms) 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6778 ms) 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6851 ms) 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (8047 ms) 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (8216 ms) 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (152605 ms total) 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: 2024-05-15T13:08:49.573 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [----------] Global test environment tear-down 2024-05-15T13:08:49.574 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (2061610 ms total) 2024-05-15T13:08:49.574 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio: [ PASSED ] 42 tests. 2024-05-15T13:08:57.390 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:26.599 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:57.393 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:26.602 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:57.395 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:26.563 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:57.398 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:26.565 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:57.400 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:26.609 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:57.401 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:26.611 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:57.485 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:26.653 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:57.493 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:26.703 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:08:57.523 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:26.732 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:02.297 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:31.506 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:02.301 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:31.511 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:02.321 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:31.530 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:02.348 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:31.516 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:02.349 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:31.517 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:02.358 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:31.567 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:02.359 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:31.569 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:02.361 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:31.529 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:02.372 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:31.582 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:17.344 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:46.553 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:17.345 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:46.555 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:17.374 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:46.584 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:17.388 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:46.556 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:17.391 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:46.559 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:17.393 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:46.603 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:17.396 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:46.606 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:17.422 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:46.590 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:17.427 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:46.637 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:18.287 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:47.455 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:18.287 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:47.456 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:18.297 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:37:47.465 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:18.328 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:47.537 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:18.329 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:47.539 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:18.341 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:37:47.550 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:18.355 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:47.565 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:18.357 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:47.566 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:18.366 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:37:47.576 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:24.656 INFO:tasks.workunit.client.0.smithi049.stdout:io.RoundTripAppendPP (2726 ms) 2024-05-15T13:09:24.656 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-05-15T13:09:24.656 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3370 ms) 2024-05-15T13:09:24.656 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-05-15T13:09:24.656 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3308 ms) 2024-05-15T13:09:24.656 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-05-15T13:09:24.656 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (4583 ms) 2024-05-15T13:09:24.656 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: setting pool EIO 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: waiting for 283 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: max_success 123, min_failed 124 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (9421 ms) 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4367 ms) 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (1960739 ms total) 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2972 ms) 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9945 ms) 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15839 ms) 2024-05-15T13:09:24.657 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3216 ms) 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (31972 ms total) 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3437 ms) 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3437 ms total) 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (15563 ms) 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6547 ms) 2024-05-15T13:09:24.658 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6982 ms) 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7485 ms) 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2515 ms) 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7816 ms) 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6941 ms) 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (8096 ms) 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7967 ms) 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-05-15T13:09:24.659 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (8091 ms) 2024-05-15T13:09:24.660 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-05-15T13:09:24.660 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7707 ms) 2024-05-15T13:09:24.660 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-05-15T13:09:24.660 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6859 ms) 2024-05-15T13:09:24.660 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-05-15T13:09:24.660 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7978 ms) 2024-05-15T13:09:32.403 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:01.571 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:32.404 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:01.572 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:32.425 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:01.634 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:32.427 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:01.637 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:32.432 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:01.600 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:32.463 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:01.672 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:32.499 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:01.708 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:32.503 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:01.712 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:32.533 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:01.742 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:33.314 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:02.482 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:33.315 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:02.483 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:33.333 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:02.501 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:33.483 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:02.692 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:33.484 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:02.693 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:33.498 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:02.707 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:33.561 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:02.771 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:33.562 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:02.772 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:33.576 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:02.786 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:37.603 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:06.813 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:37.607 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:06.816 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:37.623 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:06.833 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:37.656 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:06.824 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:37.657 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:06.825 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:37.666 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:06.875 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:37.667 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:06.876 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:37.672 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:06.840 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:37.684 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:06.893 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:47.725 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:16.934 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:47.726 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:16.936 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:47.750 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:16.960 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:47.774 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:16.942 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:47.775 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:16.944 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:47.780 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:16.990 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:47.782 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:16.991 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:47.799 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:16.967 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:09:47.805 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:17.015 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:02.736 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:31.945 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:02.743 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:31.952 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:02.790 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:31.999 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:02.801 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:31.969 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:02.811 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:31.978 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:02.814 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:32.024 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:02.821 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:32.030 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:02.841 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:32.008 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:02.856 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:32.065 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:15.086 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-05-15T13:10:15.086 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3868 ms) 2024-05-15T13:10:15.086 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-05-15T13:10:15.086 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7728 ms) 2024-05-15T13:10:15.086 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-05-15T13:10:15.086 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7754 ms) 2024-05-15T13:10:15.086 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-05-15T13:10:15.086 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7989 ms) 2024-05-15T13:10:15.086 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-05-15T13:10:15.087 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (8017 ms) 2024-05-15T13:10:15.087 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-05-15T13:10:15.087 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7223 ms) 2024-05-15T13:10:15.087 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-05-15T13:10:15.087 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7847 ms) 2024-05-15T13:10:15.087 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (150974 ms total) 2024-05-15T13:10:15.087 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: 2024-05-15T13:10:15.087 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-05-15T13:10:15.087 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (2147122 ms total) 2024-05-15T13:10:15.087 INFO:tasks.workunit.client.0.smithi049.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-05-15T13:10:15.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39925 2024-05-15T13:10:15.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 39925 2024-05-15T13:10:15.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41022 2024-05-15T13:10:15.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 41022 2024-05-15T13:10:15.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40222 2024-05-15T13:10:15.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40222 2024-05-15T13:10:15.090 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41053 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 41053 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40301 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40301 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40803 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40803 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41083 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 41083 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40242 2024-05-15T13:10:15.091 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40242 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40628 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40628 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40991 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40991 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40741 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40741 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41034 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 41034 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.092 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40526 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40526 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40096 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40096 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41066 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 41066 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40431 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40431 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.093 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39898 2024-05-15T13:10:15.094 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 39898 2024-05-15T13:10:15.094 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.094 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39953 2024-05-15T13:10:15.094 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 39953 2024-05-15T13:10:15.094 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.094 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40363 2024-05-15T13:10:15.094 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40363 2024-05-15T13:10:15.094 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.094 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=39997 2024-05-15T13:10:15.094 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 39997 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40477 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40477 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40493 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40493 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=40918 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 40918 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ for t in "${!pids[@]}" 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ pid=41117 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ wait 41117 2024-05-15T13:10:15.095 INFO:tasks.workunit.client.0.smithi049.stderr:+ exit 0 2024-05-15T13:10:15.096 INFO:tasks.workunit.client.0.smithi049.stderr:+ cleanup 2024-05-15T13:10:15.096 INFO:tasks.workunit.client.0.smithi049.stderr:+ pkill -P 39892 2024-05-15T13:10:15.100 INFO:tasks.workunit.client.0.smithi049.stderr:+ true 2024-05-15T13:10:15.100 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-15T13:10:15.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-15T13:10:15.131 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-05-15T13:10:15.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-15T13:10:15.457 DEBUG:teuthology.parallel:result is None 2024-05-15T13:10:15.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-15T13:10:15.483 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-15T13:10:15.483 DEBUG:teuthology.orchestra.run.smithi049:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-05-15T13:10:15.557 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-05-15T13:10:15.557 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-15T13:10:15.566 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-15T13:10:15.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T13:10:15.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.710+0000 7f436ac90640 10 monclient: get_monmap_and_config 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.710+0000 7f436ac90640 10 monclient: build_initial_monmap 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.710+0000 7f436ac90640 10 monclient: monmap: 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:15.711696+0000 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:15.711696+0000 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:15.701 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:15.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 1 Processor -- start 2024-05-15T13:10:15.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 1 -- start start 2024-05-15T13:10:15.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 10 monclient: init 2024-05-15T13:10:15.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:15.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f43640b1290 0x7f436406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 10 monclient(hunting): picked mon.noname-a con 0x7f43640b1290 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f436406ff90 0x7f436413c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 10 monclient(hunting): picked mon.noname-c con 0x7f436406ff90 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f436413c8b0 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4364071ff0 con 0x7f436413c8b0 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43640b1670 con 0x7f43640b1290 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f436413cd00 con 0x7f436406ff90 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:15.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.711+0000 7f436ac90640 10 monclient(hunting): authenticate will time out at 3773.874230s 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.712+0000 7f4363fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f436406ff90 0x7f436413c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.712+0000 7f4368a2c640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f43640b1290 0x7f436406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.712+0000 7f436922d640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f436413c8b0 0x7f43641428e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:55206/0 (socket says 172.21.15.49:55206) 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.712+0000 7f436922d640 1 -- 172.21.15.49:0/352502796 learned_addr learned my addr 172.21.15.49:0/352502796 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.712+0000 7f4363fff640 10 monclient(hunting): get_auth_request con 0x7f436406ff90 auth_method 0 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.712+0000 7f4363fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.712+0000 7f4363fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.712+0000 7f4363fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.713+0000 7f4368a2c640 10 monclient(hunting): get_auth_request con 0x7f43640b1290 auth_method 0 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.713+0000 7f4368a2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.713+0000 7f4368a2c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.713+0000 7f4368a2c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:15.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.713+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:15.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.713+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:15.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.713+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:15.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.713+0000 7f4368a2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:15.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.713+0000 7f4368a2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:15.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.713+0000 7f4368a2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:15.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.715+0000 7f4363fff640 10 monclient(hunting): handle_auth_done global_id 21982 payload 274 2024-05-15T13:10:15.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.715+0000 7f4363fff640 1 -- 172.21.15.49:0/352502796 >> v2:172.21.15.49:3300/0 conn(0x7f43640b1290 msgr2=0x7f436406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:15.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.715+0000 7f4363fff640 1 --2- 172.21.15.49:0/352502796 >> v2:172.21.15.49:3300/0 conn(0x7f43640b1290 0x7f436406fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:15.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.715+0000 7f4363fff640 1 -- 172.21.15.49:0/352502796 >> v1:172.21.15.49:6789/0 conn(0x7f436413c8b0 legacy=0x7f43641428e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:15.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.715+0000 7f4363fff640 10 monclient: _finish_hunting 0 2024-05-15T13:10:15.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.715+0000 7f4363fff640 1 monclient: found mon.noname-c 2024-05-15T13:10:15.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.715+0000 7f4363fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:15.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.715+0000 7f4363fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T13:10:15.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.715+0000 7f4363fff640 1 -- 172.21.15.49:0/352502796 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4364072260 con 0x7f436406ff90 2024-05-15T13:10:15.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.717+0000 7f4363fff640 1 --2- 172.21.15.49:0/352502796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f436406ff90 0x7f436413c370 secure :-1 s=READY pgs=5441 cs=0 l=1 rev1=1 crypto rx=0x7f43640b4c90 tx=0x7f435402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afd51b4ee0961363 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:15.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 1 -- 172.21.15.49:0/352502796 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43540315a0 con 0x7f436406ff90 2024-05-15T13:10:15.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 10 monclient: dump: 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 10 monclient: _finish_auth 0 2024-05-15T13:10:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 10 monclient: _check_auth_tickets 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 1 -- 172.21.15.49:0/352502796 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4354037070 con 0x7f436406ff90 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 5 monclient: authenticate success, global_id 21982 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 1 -- 172.21.15.49:0/352502796 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354031960 con 0x7f436406ff90 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 4 set_mon_vals no callback set 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f43637fe640 10 monclient: dump: 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:15.714 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:15.714 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:15.714 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:15.714 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:15.714 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:15.714 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:15.714 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:15.714 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 10 monclient: shutdown 2024-05-15T13:10:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 1 -- 172.21.15.49:0/352502796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f436406ff90 msgr2=0x7f436413c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 1 --2- 172.21.15.49:0/352502796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f436406ff90 0x7f436413c370 secure :-1 s=READY pgs=5441 cs=0 l=1 rev1=1 crypto rx=0x7f43640b4c90 tx=0x7f435402b840 comp rx=0 tx=0).stop 2024-05-15T13:10:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 1 -- 172.21.15.49:0/352502796 shutdown_connections 2024-05-15T13:10:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 1 --2- 172.21.15.49:0/352502796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f436406ff90 0x7f436413c370 unknown :-1 s=CLOSED pgs=5441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.721+0000 7f436ac90640 1 --2- 172.21.15.49:0/352502796 >> v2:172.21.15.49:3300/0 conn(0x7f43640b1290 0x7f436406fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.722+0000 7f436ac90640 1 -- 172.21.15.49:0/352502796 >> 172.21.15.49:0/352502796 conn(0x7f43640b4440 msgr2=0x7f43640b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.722+0000 7f436ac90640 1 -- 172.21.15.49:0/352502796 shutdown_connections 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.722+0000 7f436ac90640 1 -- 172.21.15.49:0/352502796 wait complete. 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.722+0000 7f436ac90640 10 monclient: build_initial_monmap 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.722+0000 7f436ac90640 10 monclient: monmap: 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:15.719 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.722+0000 7f436ac90640 1 librados: starting msgr at 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.722+0000 7f436ac90640 1 librados: starting objecter 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 1 Processor -- start 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 1 -- start start 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 1 librados: setting wanted keys 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 1 librados: calling monclient init 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient: init 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f436406ff90 0x7f436413e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient(hunting): picked mon.noname-a con 0x7f436406ff90 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43640b1290 0x7f43640a9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient(hunting): picked mon.noname-b con 0x7f43640b1290 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f436413c8b0 0x7f43640ab300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient(hunting): picked mon.noname-c con 0x7f436413c8b0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4364072f60 con 0x7f436406ff90 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43640703d0 con 0x7f43640b1290 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4364140170 con 0x7f436413c8b0 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.723+0000 7f436ac90640 10 monclient(hunting): authenticate will time out at 3773.886265s 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4368a2c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f436406ff90 0x7f436413e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f436922d640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f436413c8b0 0x7f43640ab300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4363fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43640b1290 0x7f43640a9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4368a2c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f436406ff90 0x7f436413e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59830/0 (socket says 172.21.15.49:59830) 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4368a2c640 1 -- 172.21.15.49:0/3735442537 learned_addr learned my addr 172.21.15.49:0/3735442537 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4368a2c640 10 monclient(hunting): get_auth_request con 0x7f436406ff90 auth_method 0 2024-05-15T13:10:15.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4368a2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4368a2c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4368a2c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4363fff640 10 monclient(hunting): get_auth_request con 0x7f43640b1290 auth_method 0 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4363fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4363fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f4363fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f436922d640 10 monclient(hunting): get_auth_request con 0x7f436413c8b0 auth_method 0 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f436922d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f436922d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.724+0000 7f436922d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.725+0000 7f4368a2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.725+0000 7f4368a2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.725+0000 7f4368a2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:15.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.725+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:15.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.725+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:15.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.725+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:15.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.725+0000 7f436922d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:15.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.725+0000 7f436922d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:15.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.725+0000 7f436922d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.728+0000 7f4368a2c640 10 monclient(hunting): handle_auth_done global_id 16137 payload 824 2024-05-15T13:10:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.728+0000 7f4368a2c640 1 -- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f436413c8b0 msgr2=0x7f43640ab300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.728+0000 7f4368a2c640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f436413c8b0 0x7f43640ab300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.728+0000 7f4368a2c640 1 -- 172.21.15.49:0/3735442537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43640b1290 msgr2=0x7f43640a9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.728+0000 7f4368a2c640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43640b1290 0x7f43640a9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.728+0000 7f4368a2c640 10 monclient: _finish_hunting 0 2024-05-15T13:10:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.728+0000 7f4368a2c640 1 monclient: found mon.noname-a 2024-05-15T13:10:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.728+0000 7f4368a2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.728+0000 7f4368a2c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T13:10:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.728+0000 7f4368a2c640 1 -- 172.21.15.49:0/3735442537 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43640aa920 con 0x7f436406ff90 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.730+0000 7f4368a2c640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f436406ff90 0x7f436413e450 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f434c00e9b0 tx=0x7f434c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.730+0000 7f43617fa640 10 client.?.objecter ms_handle_connect 0x7f436406ff90 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.730+0000 7f43617fa640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 1 -- 172.21.15.49:0/3735442537 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c017070 con 0x7f436406ff90 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 10 monclient: dump: 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:15.726 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 10 monclient: _finish_auth 0 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 10 monclient: _check_auth_tickets 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 1 -- 172.21.15.49:0/3735442537 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434c00c9a0 con 0x7f436406ff90 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f436ac90640 5 monclient: authenticate success, global_id 16137 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f436ac90640 10 monclient: _renew_subs 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f436ac90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f436ac90640 1 -- 172.21.15.49:0/3735442537 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43640aac00 con 0x7f436406ff90 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 1 -- 172.21.15.49:0/3735442537 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c00cd90 con 0x7f436406ff90 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:15.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f43617fa640 10 monclient: dump: 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f4362ffd640 4 set_mon_vals no callback set 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f436ac90640 10 client.16137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f436ac90640 10 monclient: _renew_subs 2024-05-15T13:10:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f436ac90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T13:10:15.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f436ac90640 1 -- 172.21.15.49:0/3735442537 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43640a9e20 con 0x7f436406ff90 2024-05-15T13:10:15.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.734+0000 7f436ac90640 1 librados: init done 2024-05-15T13:10:15.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.735+0000 7f436ac90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:15.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.735+0000 7f436ac90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:15.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.735+0000 7f436ac90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T13:10:15.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.735+0000 7f436ac90640 1 -- 172.21.15.49:0/3735442537 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436404f0e0 con 0x7f436406ff90 2024-05-15T13:10:15.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.754+0000 7f43617fa640 1 -- 172.21.15.49:0/3735442537 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f434c01a070 con 0x7f436406ff90 2024-05-15T13:10:15.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.755+0000 7f43617fa640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f434406ce90 0x7f434406f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:15.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.755+0000 7f4363fff640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f434406ce90 0x7f434406f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:15.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.755+0000 7f4363fff640 10 monclient: get_auth_request con 0x7f434406ce90 auth_method 0 2024-05-15T13:10:15.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.755+0000 7f4363fff640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f434406ce90 0x7f434406f350 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4354036000 tx=0x7f435403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:15.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.755+0000 7f43617fa640 10 client.16137.objecter ms_handle_connect 0x7f434406ce90 2024-05-15T13:10:15.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.757+0000 7f43617fa640 1 -- 172.21.15.49:0/3735442537 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1250..1250 src has 936..1250) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f434c057230 con 0x7f436406ff90 2024-05-15T13:10:15.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.757+0000 7f43617fa640 10 client.16137.objecter ms_dispatch 0x7f4364000b90 osd_map(1250..1250 src has 936..1250) v4 2024-05-15T13:10:15.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.757+0000 7f43617fa640 3 client.16137.objecter handle_osd_map got epochs [1250,1250] > 0 2024-05-15T13:10:15.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.757+0000 7f43617fa640 3 client.16137.objecter handle_osd_map decoding full epoch 1250 2024-05-15T13:10:15.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:15.757+0000 7f43617fa640 20 client.16137.objecter dump_active .. 0 homeless 2024-05-15T13:10:16.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.019+0000 7f43617fa640 1 -- 172.21.15.49:0/3735442537 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f434c059190 con 0x7f436406ff90 2024-05-15T13:10:16.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.019+0000 7f43617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:16.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.019+0000 7f43617fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:16.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.209+0000 7f436ac90640 1 -- 172.21.15.49:0/3735442537 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f43640730e0 con 0x7f434406ce90 2024-05-15T13:10:16.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.214+0000 7f43617fa640 1 -- 172.21.15.49:0/3735442537 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+197031 (secure 0 0 0) 0x7f43640730e0 con 0x7f434406ce90 2024-05-15T13:10:16.207 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T13:10:16.208 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-05-15T13:10:16.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 10 librados: watch_flush enter 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 10 librados: watch_flush exit 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 20 client.16137.objecter shutdown clearing up homeless session... 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 10 client.16137.objecter successfully canceled tick 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 -- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f434406ce90 msgr2=0x7f434406f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f434406ce90 0x7f434406f350 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4354036000 tx=0x7f435403d040 comp rx=0 tx=0).stop 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 10 monclient: shutdown 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 -- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f436406ff90 msgr2=0x7f436413e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f436406ff90 0x7f436413e450 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f434c00e9b0 tx=0x7f434c00ee70 comp rx=0 tx=0).stop 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 -- 172.21.15.49:0/3735442537 shutdown_connections 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f436413c8b0 0x7f43640ab300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f434406ce90 0x7f434406f350 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43640b1290 0x7f43640a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 --2- 172.21.15.49:0/3735442537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f436406ff90 0x7f436413e450 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 -- 172.21.15.49:0/3735442537 >> 172.21.15.49:0/3735442537 conn(0x7f43640b4440 msgr2=0x7f436412eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:16.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 -- 172.21.15.49:0/3735442537 shutdown_connections 2024-05-15T13:10:16.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 -- 172.21.15.49:0/3735442537 wait complete. 2024-05-15T13:10:16.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.218+0000 7f436ac90640 1 librados: shutdown 2024-05-15T13:10:16.223 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2387,"stamp":"2024-05-15T13:10:15.063213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465988,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":170,"num_write_kb":1429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133,"ondisk_log_size":133,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":151,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":576976,"kb_used_data":42912,"kb_used_omap":12,"kb_used_meta":533939,"kb_avail":754397744,"statfs":{"total":773094113280,"available":772503289856,"internally_reserved":0,"allocated":43941888,"data_stored":22647839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":546754096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":142,"apply_latency_ms":142,"commit_latency_ns":142000000,"apply_latency_ns":142000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012834"},"pg_stats":[{"pgid":"64.1f","version":"189'3","reported_seq":2367,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.525275+0000","last_change":"2024-05-15T12:35:50.056262+0000","last_active":"2024-05-15T13:10:09.525275+0000","last_peered":"2024-05-15T13:10:09.525275+0000","last_clean":"2024-05-15T13:10:09.525275+0000","last_became_active":"2024-05-15T12:35:49.922024+0000","last_became_peered":"2024-05-15T12:35:49.922024+0000","last_unstale":"2024-05-15T13:10:09.525275+0000","last_undegraded":"2024-05-15T13:10:09.525275+0000","last_fullsized":"2024-05-15T13:10:09.525275+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:51:48.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1e","version":"0'0","reported_seq":2366,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.766043+0000","last_change":"2024-05-15T12:35:32.331096+0000","last_active":"2024-05-15T13:10:11.766043+0000","last_peered":"2024-05-15T13:10:11.766043+0000","last_clean":"2024-05-15T13:10:11.766043+0000","last_became_active":"2024-05-15T12:35:32.314677+0000","last_became_peered":"2024-05-15T12:35:32.314677+0000","last_unstale":"2024-05-15T13:10:11.766043+0000","last_undegraded":"2024-05-15T13:10:11.766043+0000","last_fullsized":"2024-05-15T13:10:11.766043+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:14:42.373968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1d","version":"190'2","reported_seq":2356,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.302418+0000","last_change":"2024-05-15T12:36:16.192838+0000","last_active":"2024-05-15T13:10:13.302418+0000","last_peered":"2024-05-15T13:10:13.302418+0000","last_clean":"2024-05-15T13:10:13.302418+0000","last_became_active":"2024-05-15T12:36:16.182561+0000","last_became_peered":"2024-05-15T12:36:16.182561+0000","last_unstale":"2024-05-15T13:10:13.302418+0000","last_undegraded":"2024-05-15T13:10:13.302418+0000","last_fullsized":"2024-05-15T13:10:13.302418+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:22:35.686460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1c","version":"0'0","reported_seq":2363,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.491424+0000","last_change":"2024-05-15T12:35:47.611774+0000","last_active":"2024-05-15T13:10:09.491424+0000","last_peered":"2024-05-15T13:10:09.491424+0000","last_clean":"2024-05-15T13:10:09.491424+0000","last_became_active":"2024-05-15T12:35:47.561938+0000","last_became_peered":"2024-05-15T12:35:47.561938+0000","last_unstale":"2024-05-15T13:10:09.491424+0000","last_undegraded":"2024-05-15T13:10:09.491424+0000","last_fullsized":"2024-05-15T13:10:09.491424+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:53:18.771807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1b","version":"163'1","reported_seq":2375,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.806601+0000","last_change":"2024-05-15T12:35:31.343846+0000","last_active":"2024-05-15T13:10:11.806601+0000","last_peered":"2024-05-15T13:10:11.806601+0000","last_clean":"2024-05-15T13:10:11.806601+0000","last_became_active":"2024-05-15T12:35:31.337467+0000","last_became_peered":"2024-05-15T12:35:31.337467+0000","last_unstale":"2024-05-15T13:10:11.806601+0000","last_undegraded":"2024-05-15T13:10:11.806601+0000","last_fullsized":"2024-05-15T13:10:11.806601+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:43:08.695881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.1a","version":"190'4","reported_seq":2359,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.284412+0000","last_change":"2024-05-15T12:35:44.341452+0000","last_active":"2024-05-15T13:10:13.284412+0000","last_peered":"2024-05-15T13:10:13.284412+0000","last_clean":"2024-05-15T13:10:13.284412+0000","last_became_active":"2024-05-15T12:35:44.321742+0000","last_became_peered":"2024-05-15T12:35:44.321742+0000","last_unstale":"2024-05-15T13:10:13.284412+0000","last_undegraded":"2024-05-15T13:10:13.284412+0000","last_fullsized":"2024-05-15T13:10:13.284412+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:47:56.382044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.19","version":"190'1","reported_seq":2365,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.454030+0000","last_change":"2024-05-15T12:35:49.203007+0000","last_active":"2024-05-15T13:10:09.454030+0000","last_peered":"2024-05-15T13:10:09.454030+0000","last_clean":"2024-05-15T13:10:09.454030+0000","last_became_active":"2024-05-15T12:35:49.195589+0000","last_became_peered":"2024-05-15T12:35:49.195589+0000","last_unstale":"2024-05-15T13:10:09.454030+0000","last_undegraded":"2024-05-15T13:10:09.454030+0000","last_fullsized":"2024-05-15T13:10:09.454030+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:31:42.949909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.18","version":"165'3","reported_seq":2358,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.238393+0000","last_change":"2024-05-15T12:35:44.420062+0000","last_active":"2024-05-15T13:10:13.238393+0000","last_peered":"2024-05-15T13:10:13.238393+0000","last_clean":"2024-05-15T13:10:13.238393+0000","last_became_active":"2024-05-15T12:35:44.403250+0000","last_became_peered":"2024-05-15T12:35:44.403250+0000","last_unstale":"2024-05-15T13:10:13.238393+0000","last_undegraded":"2024-05-15T13:10:13.238393+0000","last_fullsized":"2024-05-15T13:10:13.238393+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:32:30.979766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.17","version":"190'2","reported_seq":2363,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.748977+0000","last_change":"2024-05-15T12:35:50.527941+0000","last_active":"2024-05-15T13:10:11.748977+0000","last_peered":"2024-05-15T13:10:11.748977+0000","last_clean":"2024-05-15T13:10:11.748977+0000","last_became_active":"2024-05-15T12:35:50.518230+0000","last_became_peered":"2024-05-15T12:35:50.518230+0000","last_unstale":"2024-05-15T13:10:11.748977+0000","last_undegraded":"2024-05-15T13:10:11.748977+0000","last_fullsized":"2024-05-15T13:10:11.748977+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:55:42.493548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.16","version":"180'4","reported_seq":2351,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.767988+0000","last_change":"2024-05-15T12:35:47.452786+0000","last_active":"2024-05-15T13:10:11.767988+0000","last_peered":"2024-05-15T13:10:11.767988+0000","last_clean":"2024-05-15T13:10:11.767988+0000","last_became_active":"2024-05-15T12:35:47.446210+0000","last_became_peered":"2024-05-15T12:35:47.446210+0000","last_unstale":"2024-05-15T13:10:11.767988+0000","last_undegraded":"2024-05-15T13:10:11.767988+0000","last_fullsized":"2024-05-15T13:10:11.767988+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:10:41.816557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.15","version":"190'2","reported_seq":2366,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.514832+0000","last_change":"2024-05-15T12:35:49.852126+0000","last_active":"2024-05-15T13:10:09.514832+0000","last_peered":"2024-05-15T13:10:09.514832+0000","last_clean":"2024-05-15T13:10:09.514832+0000","last_became_active":"2024-05-15T12:35:49.823395+0000","last_became_peered":"2024-05-15T12:35:49.823395+0000","last_unstale":"2024-05-15T13:10:09.514832+0000","last_undegraded":"2024-05-15T13:10:09.514832+0000","last_fullsized":"2024-05-15T13:10:09.514832+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:35:00.942178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.14","version":"0'0","reported_seq":2361,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.795951+0000","last_change":"2024-05-15T12:35:47.935648+0000","last_active":"2024-05-15T13:10:11.795951+0000","last_peered":"2024-05-15T13:10:11.795951+0000","last_clean":"2024-05-15T13:10:11.795951+0000","last_became_active":"2024-05-15T12:35:47.926796+0000","last_became_peered":"2024-05-15T12:35:47.926796+0000","last_unstale":"2024-05-15T13:10:11.795951+0000","last_undegraded":"2024-05-15T13:10:11.795951+0000","last_fullsized":"2024-05-15T13:10:11.795951+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:12:08.277421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.13","version":"171'1","reported_seq":2376,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.810073+0000","last_change":"2024-05-15T12:35:28.188222+0000","last_active":"2024-05-15T13:10:11.810073+0000","last_peered":"2024-05-15T13:10:11.810073+0000","last_clean":"2024-05-15T13:10:11.810073+0000","last_became_active":"2024-05-15T12:35:28.182751+0000","last_became_peered":"2024-05-15T12:35:28.182751+0000","last_unstale":"2024-05-15T13:10:11.810073+0000","last_undegraded":"2024-05-15T13:10:11.810073+0000","last_fullsized":"2024-05-15T13:10:11.810073+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:38:40.701230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.12","version":"79'1","reported_seq":2374,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.716443+0000","last_change":"2024-05-15T12:35:27.725919+0000","last_active":"2024-05-15T13:10:11.716443+0000","last_peered":"2024-05-15T13:10:11.716443+0000","last_clean":"2024-05-15T13:10:11.716443+0000","last_became_active":"2024-05-15T12:35:27.719548+0000","last_became_peered":"2024-05-15T12:35:27.719548+0000","last_unstale":"2024-05-15T13:10:11.716443+0000","last_undegraded":"2024-05-15T13:10:11.716443+0000","last_fullsized":"2024-05-15T13:10:11.716443+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:26:15.224906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.11","version":"172'1","reported_seq":2364,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.726278+0000","last_change":"2024-05-15T12:35:50.434980+0000","last_active":"2024-05-15T13:10:11.726278+0000","last_peered":"2024-05-15T13:10:11.726278+0000","last_clean":"2024-05-15T13:10:11.726278+0000","last_became_active":"2024-05-15T12:35:50.426351+0000","last_became_peered":"2024-05-15T12:35:50.426351+0000","last_unstale":"2024-05-15T13:10:11.726278+0000","last_undegraded":"2024-05-15T13:10:11.726278+0000","last_fullsized":"2024-05-15T13:10:11.726278+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:11:28.596763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.10","version":"190'3","reported_seq":2361,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.735401+0000","last_change":"2024-05-15T12:35:34.592833+0000","last_active":"2024-05-15T13:10:11.735401+0000","last_peered":"2024-05-15T13:10:11.735401+0000","last_clean":"2024-05-15T13:10:11.735401+0000","last_became_active":"2024-05-15T12:35:34.587282+0000","last_became_peered":"2024-05-15T12:35:34.587282+0000","last_unstale":"2024-05-15T13:10:11.735401+0000","last_undegraded":"2024-05-15T13:10:11.735401+0000","last_fullsized":"2024-05-15T13:10:11.735401+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:35.500859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.f","version":"155'2","reported_seq":2357,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.282099+0000","last_change":"2024-05-15T12:35:47.741135+0000","last_active":"2024-05-15T13:10:13.282099+0000","last_peered":"2024-05-15T13:10:13.282099+0000","last_clean":"2024-05-15T13:10:13.282099+0000","last_became_active":"2024-05-15T12:35:47.063782+0000","last_became_peered":"2024-05-15T12:35:47.063782+0000","last_unstale":"2024-05-15T13:10:13.282099+0000","last_undegraded":"2024-05-15T13:10:13.282099+0000","last_fullsized":"2024-05-15T13:10:13.282099+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:05:49.823797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.e","version":"174'1","reported_seq":2374,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.779790+0000","last_change":"2024-05-15T12:35:28.039423+0000","last_active":"2024-05-15T13:10:11.779790+0000","last_peered":"2024-05-15T13:10:11.779790+0000","last_clean":"2024-05-15T13:10:11.779790+0000","last_became_active":"2024-05-15T12:35:28.033694+0000","last_became_peered":"2024-05-15T12:35:28.033694+0000","last_unstale":"2024-05-15T13:10:11.779790+0000","last_undegraded":"2024-05-15T13:10:11.779790+0000","last_fullsized":"2024-05-15T13:10:11.779790+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:13:51.876959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.d","version":"175'3","reported_seq":2365,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.740170+0000","last_change":"2024-05-15T12:35:50.592101+0000","last_active":"2024-05-15T13:10:11.740170+0000","last_peered":"2024-05-15T13:10:11.740170+0000","last_clean":"2024-05-15T13:10:11.740170+0000","last_became_active":"2024-05-15T12:35:50.586234+0000","last_became_peered":"2024-05-15T12:35:50.586234+0000","last_unstale":"2024-05-15T13:10:11.740170+0000","last_undegraded":"2024-05-15T13:10:11.740170+0000","last_fullsized":"2024-05-15T13:10:11.740170+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:23:52.114251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.c","version":"190'1","reported_seq":2370,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.080944+0000","last_change":"2024-05-15T12:35:39.569380+0000","last_active":"2024-05-15T13:10:14.080944+0000","last_peered":"2024-05-15T13:10:14.080944+0000","last_clean":"2024-05-15T13:10:14.080944+0000","last_became_active":"2024-05-15T12:35:39.550911+0000","last_became_peered":"2024-05-15T12:35:39.550911+0000","last_unstale":"2024-05-15T13:10:14.080944+0000","last_undegraded":"2024-05-15T13:10:14.080944+0000","last_fullsized":"2024-05-15T13:10:14.080944+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:11:58.527351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.b","version":"186'2","reported_seq":2365,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.748051+0000","last_change":"2024-05-15T12:35:22.786550+0000","last_active":"2024-05-15T13:10:11.748051+0000","last_peered":"2024-05-15T13:10:11.748051+0000","last_clean":"2024-05-15T13:10:11.748051+0000","last_became_active":"2024-05-15T12:35:22.781126+0000","last_became_peered":"2024-05-15T12:35:22.781126+0000","last_unstale":"2024-05-15T13:10:11.748051+0000","last_undegraded":"2024-05-15T13:10:11.748051+0000","last_fullsized":"2024-05-15T13:10:11.748051+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:11:51.197065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.a","version":"170'1","reported_seq":2364,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.778652+0000","last_change":"2024-05-15T12:35:47.948588+0000","last_active":"2024-05-15T13:10:11.778652+0000","last_peered":"2024-05-15T13:10:11.778652+0000","last_clean":"2024-05-15T13:10:11.778652+0000","last_became_active":"2024-05-15T12:35:47.943168+0000","last_became_peered":"2024-05-15T12:35:47.943168+0000","last_unstale":"2024-05-15T13:10:11.778652+0000","last_undegraded":"2024-05-15T13:10:11.778652+0000","last_fullsized":"2024-05-15T13:10:11.778652+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:00:16.296438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.9","version":"190'1","reported_seq":2366,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.456092+0000","last_change":"2024-05-15T12:35:49.112002+0000","last_active":"2024-05-15T13:10:09.456092+0000","last_peered":"2024-05-15T13:10:09.456092+0000","last_clean":"2024-05-15T13:10:09.456092+0000","last_became_active":"2024-05-15T12:35:49.106669+0000","last_became_peered":"2024-05-15T12:35:49.106669+0000","last_unstale":"2024-05-15T13:10:09.456092+0000","last_undegraded":"2024-05-15T13:10:09.456092+0000","last_fullsized":"2024-05-15T13:10:09.456092+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:32:01.416983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.8","version":"0'0","reported_seq":2375,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.701935+0000","last_change":"2024-05-15T12:35:27.822672+0000","last_active":"2024-05-15T13:10:11.701935+0000","last_peered":"2024-05-15T13:10:11.701935+0000","last_clean":"2024-05-15T13:10:11.701935+0000","last_became_active":"2024-05-15T12:35:27.816334+0000","last_became_peered":"2024-05-15T12:35:27.816334+0000","last_unstale":"2024-05-15T13:10:11.701935+0000","last_undegraded":"2024-05-15T13:10:11.701935+0000","last_fullsized":"2024-05-15T13:10:11.701935+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:10:34.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.7","version":"172'1","reported_seq":2365,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.527336+0000","last_change":"2024-05-15T12:35:47.775023+0000","last_active":"2024-05-15T13:10:09.527336+0000","last_peered":"2024-05-15T13:10:09.527336+0000","last_clean":"2024-05-15T13:10:09.527336+0000","last_became_active":"2024-05-15T12:35:47.766045+0000","last_became_peered":"2024-05-15T12:35:47.766045+0000","last_unstale":"2024-05-15T13:10:09.527336+0000","last_undegraded":"2024-05-15T13:10:09.527336+0000","last_fullsized":"2024-05-15T13:10:09.527336+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:35:16.103745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.6","version":"0'0","reported_seq":2377,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.056994+0000","last_change":"2024-05-15T12:35:59.068641+0000","last_active":"2024-05-15T13:10:14.056994+0000","last_peered":"2024-05-15T13:10:14.056994+0000","last_clean":"2024-05-15T13:10:14.056994+0000","last_became_active":"2024-05-15T12:35:59.010795+0000","last_became_peered":"2024-05-15T12:35:59.010795+0000","last_unstale":"2024-05-15T13:10:14.056994+0000","last_undegraded":"2024-05-15T13:10:14.056994+0000","last_fullsized":"2024-05-15T13:10:14.056994+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:26:58.973165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.5","version":"161'1","reported_seq":2357,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.304715+0000","last_change":"2024-05-15T12:36:31.757984+0000","last_active":"2024-05-15T13:10:13.304715+0000","last_peered":"2024-05-15T13:10:13.304715+0000","last_clean":"2024-05-15T13:10:13.304715+0000","last_became_active":"2024-05-15T12:36:31.177992+0000","last_became_peered":"2024-05-15T12:36:31.177992+0000","last_unstale":"2024-05-15T13:10:13.304715+0000","last_undegraded":"2024-05-15T13:10:13.304715+0000","last_fullsized":"2024-05-15T13:10:13.304715+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:51:21.937298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.4","version":"0'0","reported_seq":2375,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.047899+0000","last_change":"2024-05-15T12:35:59.243799+0000","last_active":"2024-05-15T13:10:14.047899+0000","last_peered":"2024-05-15T13:10:14.047899+0000","last_clean":"2024-05-15T13:10:14.047899+0000","last_became_active":"2024-05-15T12:35:59.238121+0000","last_became_peered":"2024-05-15T12:35:59.238121+0000","last_unstale":"2024-05-15T13:10:14.047899+0000","last_undegraded":"2024-05-15T13:10:14.047899+0000","last_fullsized":"2024-05-15T13:10:14.047899+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:44:24.673671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.3","version":"190'4","reported_seq":2369,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.476586+0000","last_change":"2024-05-15T12:35:47.667458+0000","last_active":"2024-05-15T13:10:09.476586+0000","last_peered":"2024-05-15T13:10:09.476586+0000","last_clean":"2024-05-15T13:10:09.476586+0000","last_became_active":"2024-05-15T12:35:47.639452+0000","last_became_peered":"2024-05-15T12:35:47.639452+0000","last_unstale":"2024-05-15T13:10:09.476586+0000","last_undegraded":"2024-05-15T13:10:09.476586+0000","last_fullsized":"2024-05-15T13:10:09.476586+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:00:58.006820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":704,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.2","version":"143'1","reported_seq":2389,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.037872+0000","last_change":"2024-05-15T12:35:32.749852+0000","last_active":"2024-05-15T13:10:14.037872+0000","last_peered":"2024-05-15T13:10:14.037872+0000","last_clean":"2024-05-15T13:10:14.037872+0000","last_became_active":"2024-05-15T12:35:32.699857+0000","last_became_peered":"2024-05-15T12:35:32.699857+0000","last_unstale":"2024-05-15T13:10:14.037872+0000","last_undegraded":"2024-05-15T13:10:14.037872+0000","last_fullsized":"2024-05-15T13:10:14.037872+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:24:48.769777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1","version":"190'3","reported_seq":2360,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.298789+0000","last_change":"2024-05-15T12:36:13.620976+0000","last_active":"2024-05-15T13:10:13.298789+0000","last_peered":"2024-05-15T13:10:13.298789+0000","last_clean":"2024-05-15T13:10:13.298789+0000","last_became_active":"2024-05-15T12:36:13.614839+0000","last_became_peered":"2024-05-15T12:36:13.614839+0000","last_unstale":"2024-05-15T13:10:13.298789+0000","last_undegraded":"2024-05-15T13:10:13.298789+0000","last_fullsized":"2024-05-15T13:10:13.298789+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:01:15.571189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.0","version":"166'1","reported_seq":2352,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.741098+0000","last_change":"2024-05-15T12:35:47.277656+0000","last_active":"2024-05-15T13:10:11.741098+0000","last_peered":"2024-05-15T13:10:11.741098+0000","last_clean":"2024-05-15T13:10:11.741098+0000","last_became_active":"2024-05-15T12:35:47.271050+0000","last_became_peered":"2024-05-15T12:35:47.271050+0000","last_unstale":"2024-05-15T13:10:11.741098+0000","last_undegraded":"2024-05-15T13:10:11.741098+0000","last_fullsized":"2024-05-15T13:10:11.741098+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:12:17.443633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":2399,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.279993+0000","last_change":"2024-05-15T12:34:57.169160+0000","last_active":"2024-05-15T13:10:13.279993+0000","last_peered":"2024-05-15T13:10:13.279993+0000","last_clean":"2024-05-15T13:10:13.279993+0000","last_became_active":"2024-05-15T12:34:57.154951+0000","last_became_peered":"2024-05-15T12:34:57.154951+0000","last_unstale":"2024-05-15T13:10:13.279993+0000","last_undegraded":"2024-05-15T13:10:13.279993+0000","last_fullsized":"2024-05-15T13:10:13.279993+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:37:02.028559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":2401,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.336386+0000","last_change":"2024-05-15T12:34:54.507835+0000","last_active":"2024-05-15T13:10:13.336386+0000","last_peered":"2024-05-15T13:10:13.336386+0000","last_clean":"2024-05-15T13:10:13.336386+0000","last_became_active":"2024-05-15T12:34:54.498165+0000","last_became_peered":"2024-05-15T12:34:54.498165+0000","last_unstale":"2024-05-15T13:10:13.336386+0000","last_undegraded":"2024-05-15T13:10:13.336386+0000","last_fullsized":"2024-05-15T13:10:13.336386+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:35:01.164627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":2405,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.742356+0000","last_change":"2024-05-15T12:34:52.684394+0000","last_active":"2024-05-15T13:10:11.742356+0000","last_peered":"2024-05-15T13:10:11.742356+0000","last_clean":"2024-05-15T13:10:11.742356+0000","last_became_active":"2024-05-15T12:34:52.677394+0000","last_became_peered":"2024-05-15T12:34:52.677394+0000","last_unstale":"2024-05-15T13:10:11.742356+0000","last_undegraded":"2024-05-15T13:10:11.742356+0000","last_fullsized":"2024-05-15T13:10:11.742356+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:10:31.152782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.494523+0000","last_change":"2024-05-15T12:34:50.121401+0000","last_active":"2024-05-15T13:10:09.494523+0000","last_peered":"2024-05-15T13:10:09.494523+0000","last_clean":"2024-05-15T13:10:09.494523+0000","last_became_active":"2024-05-15T12:34:50.114215+0000","last_became_peered":"2024-05-15T12:34:50.114215+0000","last_unstale":"2024-05-15T13:10:09.494523+0000","last_undegraded":"2024-05-15T13:10:09.494523+0000","last_fullsized":"2024-05-15T13:10:09.494523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:04:16.442442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":2413,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.939553+0000","last_change":"2024-05-15T12:34:54.772141+0000","last_active":"2024-05-15T13:10:13.939553+0000","last_peered":"2024-05-15T13:10:13.939553+0000","last_clean":"2024-05-15T13:10:13.939553+0000","last_became_active":"2024-05-15T12:34:54.762972+0000","last_became_peered":"2024-05-15T12:34:54.762972+0000","last_unstale":"2024-05-15T13:10:13.939553+0000","last_undegraded":"2024-05-15T13:10:13.939553+0000","last_fullsized":"2024-05-15T13:10:13.939553+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:01:23.482313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.780442+0000","last_change":"2024-05-15T12:34:34.684239+0000","last_active":"2024-05-15T13:10:11.780442+0000","last_peered":"2024-05-15T13:10:11.780442+0000","last_clean":"2024-05-15T13:10:11.780442+0000","last_became_active":"2024-05-15T12:34:34.670411+0000","last_became_peered":"2024-05-15T12:34:34.670411+0000","last_unstale":"2024-05-15T13:10:11.780442+0000","last_undegraded":"2024-05-15T13:10:11.780442+0000","last_fullsized":"2024-05-15T13:10:11.780442+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:15:28.642471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.458171+0000","last_change":"2024-05-15T12:34:50.164567+0000","last_active":"2024-05-15T13:10:09.458171+0000","last_peered":"2024-05-15T13:10:09.458171+0000","last_clean":"2024-05-15T13:10:09.458171+0000","last_became_active":"2024-05-15T12:34:50.159068+0000","last_became_peered":"2024-05-15T12:34:50.159068+0000","last_unstale":"2024-05-15T13:10:09.458171+0000","last_undegraded":"2024-05-15T13:10:09.458171+0000","last_fullsized":"2024-05-15T13:10:09.458171+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:05:11.053203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":2414,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.449508+0000","last_change":"2024-05-15T12:34:39.918992+0000","last_active":"2024-05-15T13:10:09.449508+0000","last_peered":"2024-05-15T13:10:09.449508+0000","last_clean":"2024-05-15T13:10:09.449508+0000","last_became_active":"2024-05-15T12:34:39.907050+0000","last_became_peered":"2024-05-15T12:34:39.907050+0000","last_unstale":"2024-05-15T13:10:09.449508+0000","last_undegraded":"2024-05-15T13:10:09.449508+0000","last_fullsized":"2024-05-15T13:10:09.449508+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:10:12.279986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":2392,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.796844+0000","last_change":"2024-05-15T12:34:44.459281+0000","last_active":"2024-05-15T13:10:11.796844+0000","last_peered":"2024-05-15T13:10:11.796844+0000","last_clean":"2024-05-15T13:10:11.796844+0000","last_became_active":"2024-05-15T12:34:44.443445+0000","last_became_peered":"2024-05-15T12:34:44.443445+0000","last_unstale":"2024-05-15T13:10:11.796844+0000","last_undegraded":"2024-05-15T13:10:11.796844+0000","last_fullsized":"2024-05-15T13:10:11.796844+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:03:29.028978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":2394,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.771857+0000","last_change":"2024-05-15T12:34:38.200148+0000","last_active":"2024-05-15T13:10:11.771857+0000","last_peered":"2024-05-15T13:10:11.771857+0000","last_clean":"2024-05-15T13:10:11.771857+0000","last_became_active":"2024-05-15T12:34:38.194477+0000","last_became_peered":"2024-05-15T12:34:38.194477+0000","last_unstale":"2024-05-15T13:10:11.771857+0000","last_undegraded":"2024-05-15T13:10:11.771857+0000","last_fullsized":"2024-05-15T13:10:11.771857+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:28:19.294534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":2400,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.306918+0000","last_change":"2024-05-15T12:34:42.333270+0000","last_active":"2024-05-15T13:10:13.306918+0000","last_peered":"2024-05-15T13:10:13.306918+0000","last_clean":"2024-05-15T13:10:13.306918+0000","last_became_active":"2024-05-15T12:34:42.317054+0000","last_became_peered":"2024-05-15T12:34:42.317054+0000","last_unstale":"2024-05-15T13:10:13.306918+0000","last_undegraded":"2024-05-15T13:10:13.306918+0000","last_fullsized":"2024-05-15T13:10:13.306918+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:15:51.004019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":2411,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.078980+0000","last_change":"2024-05-15T12:34:48.752104+0000","last_active":"2024-05-15T13:10:14.078980+0000","last_peered":"2024-05-15T13:10:14.078980+0000","last_clean":"2024-05-15T13:10:14.078980+0000","last_became_active":"2024-05-15T12:34:48.742068+0000","last_became_peered":"2024-05-15T12:34:48.742068+0000","last_unstale":"2024-05-15T13:10:14.078980+0000","last_undegraded":"2024-05-15T13:10:14.078980+0000","last_fullsized":"2024-05-15T13:10:14.078980+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:18:38.517375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"26'76","reported_seq":2461,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.243120+0000","last_change":"2024-05-15T12:32:43.562093+0000","last_active":"2024-05-15T13:10:13.243120+0000","last_peered":"2024-05-15T13:10:13.243120+0000","last_clean":"2024-05-15T13:10:13.243120+0000","last_became_active":"2024-05-15T12:32:42.889644+0000","last_became_peered":"2024-05-15T12:32:42.889644+0000","last_unstale":"2024-05-15T13:10:13.243120+0000","last_undegraded":"2024-05-15T13:10:13.243120+0000","last_fullsized":"2024-05-15T13:10:13.243120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:30.812143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:30.812143+0000","last_clean_scrub_stamp":"2024-05-15T12:32:30.812143+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:45:22.988060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2420,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.718656+0000","last_change":"2024-05-15T12:32:47.722217+0000","last_active":"2024-05-15T13:10:11.718656+0000","last_peered":"2024-05-15T13:10:11.718656+0000","last_clean":"2024-05-15T13:10:11.718656+0000","last_became_active":"2024-05-15T12:32:47.716674+0000","last_became_peered":"2024-05-15T12:32:47.716674+0000","last_unstale":"2024-05-15T13:10:11.718656+0000","last_undegraded":"2024-05-15T13:10:11.718656+0000","last_fullsized":"2024-05-15T13:10:11.718656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:25:03.820946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":2413,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.530936+0000","last_change":"2024-05-15T12:34:42.947741+0000","last_active":"2024-05-15T13:10:09.530936+0000","last_peered":"2024-05-15T13:10:09.530936+0000","last_clean":"2024-05-15T13:10:09.530936+0000","last_became_active":"2024-05-15T12:34:42.942214+0000","last_became_peered":"2024-05-15T12:34:42.942214+0000","last_unstale":"2024-05-15T13:10:09.530936+0000","last_undegraded":"2024-05-15T13:10:09.530936+0000","last_fullsized":"2024-05-15T13:10:09.530936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:08:22.005787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2421,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.702999+0000","last_change":"2024-05-15T12:32:47.459103+0000","last_active":"2024-05-15T13:10:11.702999+0000","last_peered":"2024-05-15T13:10:11.702999+0000","last_clean":"2024-05-15T13:10:11.702999+0000","last_became_active":"2024-05-15T12:32:47.430194+0000","last_became_peered":"2024-05-15T12:32:47.430194+0000","last_unstale":"2024-05-15T13:10:11.702999+0000","last_undegraded":"2024-05-15T13:10:11.702999+0000","last_fullsized":"2024-05-15T13:10:11.702999+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:14:58.294854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.744310+0000","last_change":"2024-05-15T12:34:52.363872+0000","last_active":"2024-05-15T13:10:11.744310+0000","last_peered":"2024-05-15T13:10:11.744310+0000","last_clean":"2024-05-15T13:10:11.744310+0000","last_became_active":"2024-05-15T12:34:52.357328+0000","last_became_peered":"2024-05-15T12:34:52.357328+0000","last_unstale":"2024-05-15T13:10:11.744310+0000","last_undegraded":"2024-05-15T13:10:11.744310+0000","last_fullsized":"2024-05-15T13:10:11.744310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:05:48.456769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2408,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.743300+0000","last_change":"2024-05-15T12:32:47.463044+0000","last_active":"2024-05-15T13:10:11.743300+0000","last_peered":"2024-05-15T13:10:11.743300+0000","last_clean":"2024-05-15T13:10:11.743300+0000","last_became_active":"2024-05-15T12:32:47.432679+0000","last_became_peered":"2024-05-15T12:32:47.432679+0000","last_unstale":"2024-05-15T13:10:11.743300+0000","last_undegraded":"2024-05-15T13:10:11.743300+0000","last_fullsized":"2024-05-15T13:10:11.743300+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:13:56.912246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":2414,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.800112+0000","last_change":"2024-05-15T12:34:36.929836+0000","last_active":"2024-05-15T13:10:11.800112+0000","last_peered":"2024-05-15T13:10:11.800112+0000","last_clean":"2024-05-15T13:10:11.800112+0000","last_became_active":"2024-05-15T12:34:36.924450+0000","last_became_peered":"2024-05-15T12:34:36.924450+0000","last_unstale":"2024-05-15T13:10:11.800112+0000","last_undegraded":"2024-05-15T13:10:11.800112+0000","last_fullsized":"2024-05-15T13:10:11.800112+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:13:01.098128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2423,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.712047+0000","last_change":"2024-05-15T12:36:58.748207+0000","last_active":"2024-05-15T13:10:11.712047+0000","last_peered":"2024-05-15T13:10:11.712047+0000","last_clean":"2024-05-15T13:10:11.712047+0000","last_became_active":"2024-05-15T12:32:47.670226+0000","last_became_peered":"2024-05-15T12:32:47.670226+0000","last_unstale":"2024-05-15T13:10:11.712047+0000","last_undegraded":"2024-05-15T13:10:11.712047+0000","last_fullsized":"2024-05-15T13:10:11.712047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:41:09.623434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.751327+0000","last_change":"2024-05-15T12:34:51.754656+0000","last_active":"2024-05-15T13:10:11.751327+0000","last_peered":"2024-05-15T13:10:11.751327+0000","last_clean":"2024-05-15T13:10:11.751327+0000","last_became_active":"2024-05-15T12:34:51.746853+0000","last_became_peered":"2024-05-15T12:34:51.746853+0000","last_unstale":"2024-05-15T13:10:11.751327+0000","last_undegraded":"2024-05-15T13:10:11.751327+0000","last_fullsized":"2024-05-15T13:10:11.751327+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:59:19.605051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2426,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.497551+0000","last_change":"2024-05-15T12:32:47.687489+0000","last_active":"2024-05-15T13:10:09.497551+0000","last_peered":"2024-05-15T13:10:09.497551+0000","last_clean":"2024-05-15T13:10:09.497551+0000","last_became_active":"2024-05-15T12:32:47.680498+0000","last_became_peered":"2024-05-15T12:32:47.680498+0000","last_unstale":"2024-05-15T13:10:09.497551+0000","last_undegraded":"2024-05-15T13:10:09.497551+0000","last_fullsized":"2024-05-15T13:10:09.497551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:09:36.650810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"39'7","reported_seq":2427,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.048781+0000","last_change":"2024-05-15T12:34:50.381955+0000","last_active":"2024-05-15T13:10:14.048781+0000","last_peered":"2024-05-15T13:10:14.048781+0000","last_clean":"2024-05-15T13:10:14.048781+0000","last_became_active":"2024-05-15T12:34:50.371773+0000","last_became_peered":"2024-05-15T12:34:50.371773+0000","last_unstale":"2024-05-15T13:10:14.048781+0000","last_undegraded":"2024-05-15T13:10:14.048781+0000","last_fullsized":"2024-05-15T13:10:14.048781+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:40:48.381654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2419,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.746265+0000","last_change":"2024-05-15T12:32:47.531833+0000","last_active":"2024-05-15T13:10:11.746265+0000","last_peered":"2024-05-15T13:10:11.746265+0000","last_clean":"2024-05-15T13:10:11.746265+0000","last_became_active":"2024-05-15T12:32:47.526275+0000","last_became_peered":"2024-05-15T12:32:47.526275+0000","last_unstale":"2024-05-15T13:10:11.746265+0000","last_undegraded":"2024-05-15T13:10:11.746265+0000","last_fullsized":"2024-05-15T13:10:11.746265+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:32:09.483290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":2404,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.241050+0000","last_change":"2024-05-15T12:34:38.895220+0000","last_active":"2024-05-15T13:10:13.241050+0000","last_peered":"2024-05-15T13:10:13.241050+0000","last_clean":"2024-05-15T13:10:13.241050+0000","last_became_active":"2024-05-15T12:34:38.888318+0000","last_became_peered":"2024-05-15T12:34:38.888318+0000","last_unstale":"2024-05-15T13:10:13.241050+0000","last_undegraded":"2024-05-15T13:10:13.241050+0000","last_fullsized":"2024-05-15T13:10:13.241050+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:38:34.358044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2424,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.818412+0000","last_change":"2024-05-15T12:32:48.309674+0000","last_active":"2024-05-15T13:10:11.818412+0000","last_peered":"2024-05-15T13:10:11.818412+0000","last_clean":"2024-05-15T13:10:11.818412+0000","last_became_active":"2024-05-15T12:32:48.283635+0000","last_became_peered":"2024-05-15T12:32:48.283635+0000","last_unstale":"2024-05-15T13:10:11.818412+0000","last_undegraded":"2024-05-15T13:10:11.818412+0000","last_fullsized":"2024-05-15T13:10:11.818412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:07:32.002393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":2414,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.480560+0000","last_change":"2024-05-15T12:34:48.672942+0000","last_active":"2024-05-15T13:10:09.480560+0000","last_peered":"2024-05-15T13:10:09.480560+0000","last_clean":"2024-05-15T13:10:09.480560+0000","last_became_active":"2024-05-15T12:34:48.664879+0000","last_became_peered":"2024-05-15T12:34:48.664879+0000","last_unstale":"2024-05-15T13:10:09.480560+0000","last_undegraded":"2024-05-15T13:10:09.480560+0000","last_fullsized":"2024-05-15T13:10:09.480560+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:48:54.955944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2418,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.753278+0000","last_change":"2024-05-15T12:32:48.258451+0000","last_active":"2024-05-15T13:10:11.753278+0000","last_peered":"2024-05-15T13:10:11.753278+0000","last_clean":"2024-05-15T13:10:11.753278+0000","last_became_active":"2024-05-15T12:32:48.252725+0000","last_became_peered":"2024-05-15T12:32:48.252725+0000","last_unstale":"2024-05-15T13:10:11.753278+0000","last_undegraded":"2024-05-15T13:10:11.753278+0000","last_fullsized":"2024-05-15T13:10:11.753278+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:01:52.775805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":2414,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.782737+0000","last_change":"2024-05-15T12:34:34.676904+0000","last_active":"2024-05-15T13:10:11.782737+0000","last_peered":"2024-05-15T13:10:11.782737+0000","last_clean":"2024-05-15T13:10:11.782737+0000","last_became_active":"2024-05-15T12:34:34.660678+0000","last_became_peered":"2024-05-15T12:34:34.660678+0000","last_unstale":"2024-05-15T13:10:11.782737+0000","last_undegraded":"2024-05-15T13:10:11.782737+0000","last_fullsized":"2024-05-15T13:10:11.782737+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:39:07.124084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":2394,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.739089+0000","last_change":"2024-05-15T12:34:44.299228+0000","last_active":"2024-05-15T13:10:11.739089+0000","last_peered":"2024-05-15T13:10:11.739089+0000","last_clean":"2024-05-15T13:10:11.739089+0000","last_became_active":"2024-05-15T12:34:44.277750+0000","last_became_peered":"2024-05-15T12:34:44.277750+0000","last_unstale":"2024-05-15T13:10:11.739089+0000","last_undegraded":"2024-05-15T13:10:11.739089+0000","last_fullsized":"2024-05-15T13:10:11.739089+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:00:02.340970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":2412,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.046745+0000","last_change":"2024-05-15T12:34:39.506099+0000","last_active":"2024-05-15T13:10:14.046745+0000","last_peered":"2024-05-15T13:10:14.046745+0000","last_clean":"2024-05-15T13:10:14.046745+0000","last_became_active":"2024-05-15T12:34:39.369183+0000","last_became_peered":"2024-05-15T12:34:39.369183+0000","last_unstale":"2024-05-15T13:10:14.046745+0000","last_undegraded":"2024-05-15T13:10:14.046745+0000","last_fullsized":"2024-05-15T13:10:14.046745+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:41:03.408587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":2402,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.286633+0000","last_change":"2024-05-15T12:34:38.831156+0000","last_active":"2024-05-15T13:10:13.286633+0000","last_peered":"2024-05-15T13:10:13.286633+0000","last_clean":"2024-05-15T13:10:13.286633+0000","last_became_active":"2024-05-15T12:34:38.812314+0000","last_became_peered":"2024-05-15T12:34:38.812314+0000","last_unstale":"2024-05-15T13:10:13.286633+0000","last_undegraded":"2024-05-15T13:10:13.286633+0000","last_fullsized":"2024-05-15T13:10:13.286633+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:45:31.161837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":2393,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.745771+0000","last_change":"2024-05-15T12:34:36.784260+0000","last_active":"2024-05-15T13:10:11.745771+0000","last_peered":"2024-05-15T13:10:11.745771+0000","last_clean":"2024-05-15T13:10:11.745771+0000","last_became_active":"2024-05-15T12:34:36.777340+0000","last_became_peered":"2024-05-15T12:34:36.777340+0000","last_unstale":"2024-05-15T13:10:11.745771+0000","last_undegraded":"2024-05-15T13:10:11.745771+0000","last_fullsized":"2024-05-15T13:10:11.745771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:27:38.620258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":2410,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.077032+0000","last_change":"2024-05-15T12:34:51.056513+0000","last_active":"2024-05-15T13:10:14.077032+0000","last_peered":"2024-05-15T13:10:14.077032+0000","last_clean":"2024-05-15T13:10:14.077032+0000","last_became_active":"2024-05-15T12:34:51.040048+0000","last_became_peered":"2024-05-15T12:34:51.040048+0000","last_unstale":"2024-05-15T13:10:14.077032+0000","last_undegraded":"2024-05-15T13:10:14.077032+0000","last_fullsized":"2024-05-15T13:10:14.077032+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:32:15.066105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":2412,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.516901+0000","last_change":"2024-05-15T12:34:51.656110+0000","last_active":"2024-05-15T13:10:09.516901+0000","last_peered":"2024-05-15T13:10:09.516901+0000","last_clean":"2024-05-15T13:10:09.516901+0000","last_became_active":"2024-05-15T12:34:51.649910+0000","last_became_peered":"2024-05-15T12:34:51.649910+0000","last_unstale":"2024-05-15T13:10:09.516901+0000","last_undegraded":"2024-05-15T13:10:09.516901+0000","last_fullsized":"2024-05-15T13:10:09.516901+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:01:47.425779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":2396,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.770003+0000","last_change":"2024-05-15T12:34:44.408851+0000","last_active":"2024-05-15T13:10:11.770003+0000","last_peered":"2024-05-15T13:10:11.770003+0000","last_clean":"2024-05-15T13:10:11.770003+0000","last_became_active":"2024-05-15T12:34:44.399991+0000","last_became_peered":"2024-05-15T12:34:44.399991+0000","last_unstale":"2024-05-15T13:10:11.770003+0000","last_undegraded":"2024-05-15T13:10:11.770003+0000","last_fullsized":"2024-05-15T13:10:11.770003+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:40:47.114950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":2424,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.064927+0000","last_change":"2024-05-15T12:34:38.557851+0000","last_active":"2024-05-15T13:10:14.064927+0000","last_peered":"2024-05-15T13:10:14.064927+0000","last_clean":"2024-05-15T13:10:14.064927+0000","last_became_active":"2024-05-15T12:34:38.549385+0000","last_became_peered":"2024-05-15T12:34:38.549385+0000","last_unstale":"2024-05-15T13:10:14.064927+0000","last_undegraded":"2024-05-15T13:10:14.064927+0000","last_fullsized":"2024-05-15T13:10:14.064927+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:31:24.444568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":2411,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.756882+0000","last_change":"2024-05-15T12:34:36.889179+0000","last_active":"2024-05-15T13:10:11.756882+0000","last_peered":"2024-05-15T13:10:11.756882+0000","last_clean":"2024-05-15T13:10:11.756882+0000","last_became_active":"2024-05-15T12:34:36.875596+0000","last_became_peered":"2024-05-15T12:34:36.875596+0000","last_unstale":"2024-05-15T13:10:11.756882+0000","last_undegraded":"2024-05-15T13:10:11.756882+0000","last_fullsized":"2024-05-15T13:10:11.756882+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:30:10.134778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.723668+0000","last_change":"2024-05-15T12:34:51.659524+0000","last_active":"2024-05-15T13:10:11.723668+0000","last_peered":"2024-05-15T13:10:11.723668+0000","last_clean":"2024-05-15T13:10:11.723668+0000","last_became_active":"2024-05-15T12:34:51.653558+0000","last_became_peered":"2024-05-15T12:34:51.653558+0000","last_unstale":"2024-05-15T13:10:11.723668+0000","last_undegraded":"2024-05-15T13:10:11.723668+0000","last_fullsized":"2024-05-15T13:10:11.723668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:43:01.117943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.710084+0000","last_change":"2024-05-15T12:34:36.637973+0000","last_active":"2024-05-15T13:10:11.710084+0000","last_peered":"2024-05-15T13:10:11.710084+0000","last_clean":"2024-05-15T13:10:11.710084+0000","last_became_active":"2024-05-15T12:34:36.629372+0000","last_became_peered":"2024-05-15T12:34:36.629372+0000","last_unstale":"2024-05-15T13:10:11.710084+0000","last_undegraded":"2024-05-15T13:10:11.710084+0000","last_fullsized":"2024-05-15T13:10:11.710084+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:10:32.079108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":2408,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.812762+0000","last_change":"2024-05-15T12:34:49.358758+0000","last_active":"2024-05-15T13:10:11.812762+0000","last_peered":"2024-05-15T13:10:11.812762+0000","last_clean":"2024-05-15T13:10:11.812762+0000","last_became_active":"2024-05-15T12:34:49.324761+0000","last_became_peered":"2024-05-15T12:34:49.324761+0000","last_unstale":"2024-05-15T13:10:11.812762+0000","last_undegraded":"2024-05-15T13:10:11.812762+0000","last_fullsized":"2024-05-15T13:10:11.812762+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:13:55.863763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":6703,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":13406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313668,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68824,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303016,"statfs":{"total":96636764160,"available":96566288384,"internally_reserved":0,"allocated":5840896,"data_stored":3135087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]},{"osd":7,"up_from":19,"seq":81604379073,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72528,"kb_used_data":5252,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299312,"statfs":{"total":96636764160,"available":96562495488,"internally_reserved":0,"allocated":5378048,"data_stored":2731941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":19,"seq":81604379079,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72916,"kb_used_data":5636,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298924,"statfs":{"total":96636764160,"available":96562098176,"internally_reserved":0,"allocated":5771264,"data_stored":3110480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444485,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76712,"kb_used_data":5272,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295128,"statfs":{"total":96636764160,"available":96558211072,"internally_reserved":0,"allocated":5398528,"data_stored":2735851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444483,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72528,"kb_used_data":5248,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299312,"statfs":{"total":96636764160,"available":96562495488,"internally_reserved":0,"allocated":5373952,"data_stored":2742021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542599,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72580,"kb_used_data":5304,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299260,"statfs":{"total":96636764160,"available":96562442240,"internally_reserved":0,"allocated":5431296,"data_stored":2744265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509889,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68360,"kb_used_data":5244,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303480,"statfs":{"total":96636764160,"available":96566763520,"internally_reserved":0,"allocated":5369856,"data_stored":2720297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575301,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72528,"kb_used_data":5252,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299312,"statfs":{"total":96636764160,"available":96562495488,"internally_reserved":0,"allocated":5378048,"data_stored":2727897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:10:16.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T13:10:16.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.388+0000 7f98f8e33640 10 monclient: get_monmap_and_config 2024-05-15T13:10:16.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.388+0000 7f98f8e33640 10 monclient: build_initial_monmap 2024-05-15T13:10:16.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.388+0000 7f98f8e33640 10 monclient: monmap: 2024-05-15T13:10:16.379 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:16.379 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:16.379 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:16.389575+0000 2024-05-15T13:10:16.379 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:16.389575+0000 2024-05-15T13:10:16.379 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:16.379 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:16.379 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:16.380 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:16.380 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:16.380 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:16.380 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:16.380 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:16.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 1 Processor -- start 2024-05-15T13:10:16.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 1 -- start start 2024-05-15T13:10:16.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 10 monclient: init 2024-05-15T13:10:16.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:16.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-05-15T13:10:16.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98f40b5410 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:16.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98f41400a0 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 0x7f98f40a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 10 monclient(hunting): picked mon.noname-b con 0x7f98f4146310 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f40ab470 con 0x7f98f40b5410 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f40aa0c0 con 0x7f98f41400a0 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f40b57f0 con 0x7f98f4146310 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.389+0000 7f98f8e33640 10 monclient(hunting): authenticate will time out at 3774.552412s 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.390+0000 7f98f2d9d640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 0x7f98f40a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.390+0000 7f98f259c640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f98f40b5410 0x7f98f413f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:55212/0 (socket says 172.21.15.49:55212) 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.390+0000 7f98f259c640 1 -- 172.21.15.49:0/404089415 learned_addr learned my addr 172.21.15.49:0/404089415 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.391+0000 7f98f2d9d640 10 monclient(hunting): get_auth_request con 0x7f98f4146310 auth_method 0 2024-05-15T13:10:16.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.391+0000 7f98f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:16.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.391+0000 7f98f2d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:16.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.391+0000 7f98f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:16.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.391+0000 7f98f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:16.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.391+0000 7f98f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:16.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.391+0000 7f98f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:16.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.393+0000 7f98f1d9b640 1 -- 172.21.15.49:0/404089415 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628915018 0 0) 0x7f98f40ab470 con 0x7f98f40b5410 2024-05-15T13:10:16.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.393+0000 7f98f1d9b640 10 monclient(hunting): _negotiate 2024-05-15T13:10:16.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.393+0000 7f98f1d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:16.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.393+0000 7f98f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:16.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.393+0000 7f98f1d9b640 10 monclient(hunting): my global_id is 16140 2024-05-15T13:10:16.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.393+0000 7f98f1d9b640 1 -- 172.21.15.49:0/404089415 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e0003520 con 0x7f98f40b5410 2024-05-15T13:10:16.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.393+0000 7f98f1d9b640 1 -- 172.21.15.49:0/404089415 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987702145 0 0) 0x7f98f40aa0c0 con 0x7f98f41400a0 2024-05-15T13:10:16.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f1d9b640 10 monclient(hunting): _negotiate 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f1d9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f1d9b640 10 monclient(hunting): my global_id is 21988 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f1d9b640 1 -- 172.21.15.49:0/404089415 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f40ab470 con 0x7f98f41400a0 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f2d9d640 10 monclient(hunting): handle_auth_done global_id 17591 payload 274 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f2d9d640 1 -- 172.21.15.49:0/404089415 >> v1:172.21.15.88:6789/0 conn(0x7f98f41400a0 legacy=0x7f98f4145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f2d9d640 1 -- 172.21.15.49:0/404089415 >> v1:172.21.15.49:6789/0 conn(0x7f98f40b5410 legacy=0x7f98f413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f2d9d640 10 monclient: _finish_hunting 0 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f2d9d640 1 monclient: found mon.noname-b 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T13:10:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.394+0000 7f98f2d9d640 1 -- 172.21.15.49:0/404089415 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f40aa5e0 con 0x7f98f4146310 2024-05-15T13:10:16.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.397+0000 7f98f2d9d640 1 --2- 172.21.15.49:0/404089415 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 0x7f98f40a9960 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009dc0 tx=0x7f98ec02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2e2b61b38bc81ba server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:16.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 1 -- 172.21.15.49:0/404089415 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec039070 con 0x7f98f4146310 2024-05-15T13:10:16.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:16.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:16.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 10 monclient: dump: 2024-05-15T13:10:16.393 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:16.393 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:16.393 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 10 monclient: _finish_auth 0 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 10 monclient: _check_auth_tickets 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 1 -- 172.21.15.49:0/404089415 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec02cc40 con 0x7f98f4146310 2024-05-15T13:10:16.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f8e33640 5 monclient: authenticate success, global_id 17591 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f8e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f8e33640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 1 -- 172.21.15.49:0/404089415 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec034440 con 0x7f98f4146310 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f8e33640 4 set_mon_vals no callback set 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.402+0000 7f98f1d9b640 10 monclient: dump: 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:16.396 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:16.396 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:16.396 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:16.396 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:16.396 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:16.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.403+0000 7f98f8e33640 10 monclient: shutdown 2024-05-15T13:10:16.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.403+0000 7f98f8e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:16.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.403+0000 7f98f8e33640 1 -- 172.21.15.49:0/404089415 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 msgr2=0x7f98f40a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:16.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.403+0000 7f98f8e33640 1 --2- 172.21.15.49:0/404089415 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 0x7f98f40a9960 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009dc0 tx=0x7f98ec02be90 comp rx=0 tx=0).stop 2024-05-15T13:10:16.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.403+0000 7f98f8e33640 1 -- 172.21.15.49:0/404089415 shutdown_connections 2024-05-15T13:10:16.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.403+0000 7f98f8e33640 1 --2- 172.21.15.49:0/404089415 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 0x7f98f40a9960 unknown :-1 s=CLOSED pgs=3117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.403+0000 7f98f8e33640 1 -- 172.21.15.49:0/404089415 >> 172.21.15.49:0/404089415 conn(0x7f98f4066f00 msgr2=0x7f98f4067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:16.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.403+0000 7f98f8e33640 1 -- 172.21.15.49:0/404089415 shutdown_connections 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.403+0000 7f98f8e33640 1 -- 172.21.15.49:0/404089415 wait complete. 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.404+0000 7f98f8e33640 10 monclient: build_initial_monmap 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.404+0000 7f98f8e33640 10 monclient: monmap: 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.405+0000 7f98f8e33640 1 librados: starting msgr at 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.405+0000 7f98f8e33640 1 librados: starting objecter 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.405+0000 7f98f8e33640 1 Processor -- start 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.405+0000 7f98f8e33640 1 -- start start 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.405+0000 7f98f8e33640 1 librados: setting wanted keys 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.405+0000 7f98f8e33640 1 librados: calling monclient init 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.405+0000 7f98f8e33640 10 monclient: init 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:10:16.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98f40b5410 0x7f98f4131260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 10 monclient(hunting): picked mon.noname-b con 0x7f98f40b5410 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98f41400a0 0x7f98f41317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 10 monclient(hunting): picked mon.noname-a con 0x7f98f41400a0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 0x7f98f41db630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 10 monclient(hunting): picked mon.noname-c con 0x7f98f4146310 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f414df40 con 0x7f98f41400a0 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f40a9ea0 con 0x7f98f40b5410 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:16.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f414e0c0 con 0x7f98f4146310 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.406+0000 7f98f8e33640 10 monclient(hunting): authenticate will time out at 3774.569277s 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98f40b5410 0x7f98f4131260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98f40b5410 0x7f98f4131260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43048/0 (socket says 172.21.15.49:43048) 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98ebfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98f41400a0 0x7f98f41317a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f259c640 1 -- 172.21.15.49:0/349489045 learned_addr learned my addr 172.21.15.49:0/349489045 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f259c640 10 monclient(hunting): get_auth_request con 0x7f98f40b5410 auth_method 0 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f259c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98ebfff640 10 monclient(hunting): get_auth_request con 0x7f98f41400a0 auth_method 0 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f2d9d640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 0x7f98f41db630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98ebfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.407+0000 7f98f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.408+0000 7f98f2d9d640 10 monclient(hunting): get_auth_request con 0x7f98f4146310 auth_method 0 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.408+0000 7f98f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.408+0000 7f98f2d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.408+0000 7f98f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.408+0000 7f98ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.408+0000 7f98ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:16.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.408+0000 7f98ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:16.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.408+0000 7f98f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:16.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.408+0000 7f98f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:16.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.408+0000 7f98f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:16.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.410+0000 7f98f259c640 10 monclient(hunting): handle_auth_done global_id 21991 payload 824 2024-05-15T13:10:16.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.410+0000 7f98f259c640 1 -- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 msgr2=0x7f98f41db630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:16.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.410+0000 7f98f259c640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 0x7f98f41db630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.410+0000 7f98f259c640 1 -- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98f41400a0 msgr2=0x7f98f41317a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:16.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.410+0000 7f98f259c640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98f41400a0 0x7f98f41317a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.410+0000 7f98f259c640 10 monclient: _finish_hunting 0 2024-05-15T13:10:16.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.410+0000 7f98f259c640 1 monclient: found mon.noname-b 2024-05-15T13:10:16.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.410+0000 7f98f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:16.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.410+0000 7f98f259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.410+0000 7f98f259c640 1 -- 172.21.15.49:0/349489045 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f4140500 con 0x7f98f40b5410 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.414+0000 7f98eaffd640 10 client.?.objecter ms_handle_connect 0x7f98f40b5410 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.414+0000 7f98eaffd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.414+0000 7f98f259c640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98f40b5410 0x7f98f4131260 secure :-1 s=READY pgs=5443 cs=0 l=1 rev1=1 crypto rx=0x7f98e400c530 tx=0x7f98e400c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 1 -- 172.21.15.49:0/349489045 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e4017070 con 0x7f98f40b5410 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 10 monclient: dump: 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 10 monclient: _finish_auth 0 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 10 monclient: _check_auth_tickets 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 1 -- 172.21.15.49:0/349489045 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e4005d00 con 0x7f98f40b5410 2024-05-15T13:10:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 1 -- 172.21.15.49:0/349489045 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e40072b0 con 0x7f98f40b5410 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 5 monclient: authenticate success, global_id 21991 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 10 monclient: _renew_subs 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 1 -- 172.21.15.49:0/349489045 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f40b7190 con 0x7f98f40b5410 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98eaffd640 10 monclient: dump: 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:16.411 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f159a640 4 set_mon_vals no callback set 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 10 client.21991.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 10 monclient: _renew_subs 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 1 -- 172.21.15.49:0/349489045 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f4142c70 con 0x7f98f40b5410 2024-05-15T13:10:16.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 1 librados: init done 2024-05-15T13:10:16.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:16.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:16.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:16.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.417+0000 7f98f8e33640 1 -- 172.21.15.49:0/349489045 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f404f0e0 con 0x7f98f40b5410 2024-05-15T13:10:16.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.437+0000 7f98eaffd640 1 -- 172.21.15.49:0/349489045 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f98e401a070 con 0x7f98f40b5410 2024-05-15T13:10:16.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.438+0000 7f98eaffd640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f98d406ce90 0x7f98d406f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:16.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.438+0000 7f98ebfff640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f98d406ce90 0x7f98d406f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:16.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.438+0000 7f98ebfff640 10 monclient: get_auth_request con 0x7f98d406ce90 auth_method 0 2024-05-15T13:10:16.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.438+0000 7f98eaffd640 10 client.21991.objecter ms_handle_connect 0x7f98d406ce90 2024-05-15T13:10:16.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.438+0000 7f98ebfff640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f98d406ce90 0x7f98d406f350 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f98f4142a40 tx=0x7f98dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:16.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.439+0000 7f98eaffd640 1 -- 172.21.15.49:0/349489045 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1250..1250 src has 936..1250) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f98e4056230 con 0x7f98f40b5410 2024-05-15T13:10:16.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.439+0000 7f98eaffd640 10 client.21991.objecter ms_dispatch 0x7f98f4000b90 osd_map(1250..1250 src has 936..1250) v4 2024-05-15T13:10:16.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.439+0000 7f98eaffd640 3 client.21991.objecter handle_osd_map got epochs [1250,1250] > 0 2024-05-15T13:10:16.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.439+0000 7f98eaffd640 3 client.21991.objecter handle_osd_map decoding full epoch 1250 2024-05-15T13:10:16.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.439+0000 7f98eaffd640 20 client.21991.objecter dump_active .. 0 homeless 2024-05-15T13:10:16.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.673+0000 7f98eaffd640 1 -- 172.21.15.49:0/349489045 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f98e4021020 con 0x7f98f40b5410 2024-05-15T13:10:16.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.673+0000 7f98eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:16.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.673+0000 7f98eaffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:16.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.859+0000 7f98f8e33640 1 -- 172.21.15.49:0/349489045 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98f40b7590 con 0x7f98d406ce90 2024-05-15T13:10:16.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.863+0000 7f98eaffd640 1 -- 172.21.15.49:0/349489045 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+197031 (secure 0 0 0) 0x7f98dc0014d0 con 0x7f98d406ce90 2024-05-15T13:10:16.855 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T13:10:16.856 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-05-15T13:10:16.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 10 librados: watch_flush enter 2024-05-15T13:10:16.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 10 librados: watch_flush exit 2024-05-15T13:10:16.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 20 client.21991.objecter shutdown clearing up homeless session... 2024-05-15T13:10:16.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 10 client.21991.objecter successfully canceled tick 2024-05-15T13:10:16.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 -- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f98d406ce90 msgr2=0x7f98d406f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:16.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f98d406ce90 0x7f98d406f350 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f98f4142a40 tx=0x7f98dc009040 comp rx=0 tx=0).stop 2024-05-15T13:10:16.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 10 monclient: shutdown 2024-05-15T13:10:16.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:16.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 -- 172.21.15.49:0/349489045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98f40b5410 msgr2=0x7f98f4131260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98f40b5410 0x7f98f4131260 secure :-1 s=READY pgs=5443 cs=0 l=1 rev1=1 crypto rx=0x7f98e400c530 tx=0x7f98e400c9f0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 -- 172.21.15.49:0/349489045 shutdown_connections 2024-05-15T13:10:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98f4146310 0x7f98f41db630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98f41400a0 0x7f98f41317a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f98d406ce90 0x7f98d406f350 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 --2- 172.21.15.49:0/349489045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98f40b5410 0x7f98f4131260 unknown :-1 s=CLOSED pgs=5443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 -- 172.21.15.49:0/349489045 >> 172.21.15.49:0/349489045 conn(0x7f98f4066f00 msgr2=0x7f98f413f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 -- 172.21.15.49:0/349489045 shutdown_connections 2024-05-15T13:10:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 -- 172.21.15.49:0/349489045 wait complete. 2024-05-15T13:10:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:16.866+0000 7f98f8e33640 1 librados: shutdown 2024-05-15T13:10:16.867 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2387,"stamp":"2024-05-15T13:10:15.063213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465988,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":170,"num_write_kb":1429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133,"ondisk_log_size":133,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":151,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":576976,"kb_used_data":42912,"kb_used_omap":12,"kb_used_meta":533939,"kb_avail":754397744,"statfs":{"total":773094113280,"available":772503289856,"internally_reserved":0,"allocated":43941888,"data_stored":22647839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":546754096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":142,"apply_latency_ms":142,"commit_latency_ns":142000000,"apply_latency_ns":142000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012834"},"pg_stats":[{"pgid":"64.1f","version":"189'3","reported_seq":2367,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.525275+0000","last_change":"2024-05-15T12:35:50.056262+0000","last_active":"2024-05-15T13:10:09.525275+0000","last_peered":"2024-05-15T13:10:09.525275+0000","last_clean":"2024-05-15T13:10:09.525275+0000","last_became_active":"2024-05-15T12:35:49.922024+0000","last_became_peered":"2024-05-15T12:35:49.922024+0000","last_unstale":"2024-05-15T13:10:09.525275+0000","last_undegraded":"2024-05-15T13:10:09.525275+0000","last_fullsized":"2024-05-15T13:10:09.525275+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:51:48.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1e","version":"0'0","reported_seq":2366,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.766043+0000","last_change":"2024-05-15T12:35:32.331096+0000","last_active":"2024-05-15T13:10:11.766043+0000","last_peered":"2024-05-15T13:10:11.766043+0000","last_clean":"2024-05-15T13:10:11.766043+0000","last_became_active":"2024-05-15T12:35:32.314677+0000","last_became_peered":"2024-05-15T12:35:32.314677+0000","last_unstale":"2024-05-15T13:10:11.766043+0000","last_undegraded":"2024-05-15T13:10:11.766043+0000","last_fullsized":"2024-05-15T13:10:11.766043+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:14:42.373968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1d","version":"190'2","reported_seq":2356,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.302418+0000","last_change":"2024-05-15T12:36:16.192838+0000","last_active":"2024-05-15T13:10:13.302418+0000","last_peered":"2024-05-15T13:10:13.302418+0000","last_clean":"2024-05-15T13:10:13.302418+0000","last_became_active":"2024-05-15T12:36:16.182561+0000","last_became_peered":"2024-05-15T12:36:16.182561+0000","last_unstale":"2024-05-15T13:10:13.302418+0000","last_undegraded":"2024-05-15T13:10:13.302418+0000","last_fullsized":"2024-05-15T13:10:13.302418+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:22:35.686460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1c","version":"0'0","reported_seq":2363,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.491424+0000","last_change":"2024-05-15T12:35:47.611774+0000","last_active":"2024-05-15T13:10:09.491424+0000","last_peered":"2024-05-15T13:10:09.491424+0000","last_clean":"2024-05-15T13:10:09.491424+0000","last_became_active":"2024-05-15T12:35:47.561938+0000","last_became_peered":"2024-05-15T12:35:47.561938+0000","last_unstale":"2024-05-15T13:10:09.491424+0000","last_undegraded":"2024-05-15T13:10:09.491424+0000","last_fullsized":"2024-05-15T13:10:09.491424+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:53:18.771807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1b","version":"163'1","reported_seq":2375,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.806601+0000","last_change":"2024-05-15T12:35:31.343846+0000","last_active":"2024-05-15T13:10:11.806601+0000","last_peered":"2024-05-15T13:10:11.806601+0000","last_clean":"2024-05-15T13:10:11.806601+0000","last_became_active":"2024-05-15T12:35:31.337467+0000","last_became_peered":"2024-05-15T12:35:31.337467+0000","last_unstale":"2024-05-15T13:10:11.806601+0000","last_undegraded":"2024-05-15T13:10:11.806601+0000","last_fullsized":"2024-05-15T13:10:11.806601+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:43:08.695881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.1a","version":"190'4","reported_seq":2359,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.284412+0000","last_change":"2024-05-15T12:35:44.341452+0000","last_active":"2024-05-15T13:10:13.284412+0000","last_peered":"2024-05-15T13:10:13.284412+0000","last_clean":"2024-05-15T13:10:13.284412+0000","last_became_active":"2024-05-15T12:35:44.321742+0000","last_became_peered":"2024-05-15T12:35:44.321742+0000","last_unstale":"2024-05-15T13:10:13.284412+0000","last_undegraded":"2024-05-15T13:10:13.284412+0000","last_fullsized":"2024-05-15T13:10:13.284412+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:47:56.382044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.19","version":"190'1","reported_seq":2365,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.454030+0000","last_change":"2024-05-15T12:35:49.203007+0000","last_active":"2024-05-15T13:10:09.454030+0000","last_peered":"2024-05-15T13:10:09.454030+0000","last_clean":"2024-05-15T13:10:09.454030+0000","last_became_active":"2024-05-15T12:35:49.195589+0000","last_became_peered":"2024-05-15T12:35:49.195589+0000","last_unstale":"2024-05-15T13:10:09.454030+0000","last_undegraded":"2024-05-15T13:10:09.454030+0000","last_fullsized":"2024-05-15T13:10:09.454030+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:31:42.949909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.18","version":"165'3","reported_seq":2358,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.238393+0000","last_change":"2024-05-15T12:35:44.420062+0000","last_active":"2024-05-15T13:10:13.238393+0000","last_peered":"2024-05-15T13:10:13.238393+0000","last_clean":"2024-05-15T13:10:13.238393+0000","last_became_active":"2024-05-15T12:35:44.403250+0000","last_became_peered":"2024-05-15T12:35:44.403250+0000","last_unstale":"2024-05-15T13:10:13.238393+0000","last_undegraded":"2024-05-15T13:10:13.238393+0000","last_fullsized":"2024-05-15T13:10:13.238393+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:32:30.979766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.17","version":"190'2","reported_seq":2363,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.748977+0000","last_change":"2024-05-15T12:35:50.527941+0000","last_active":"2024-05-15T13:10:11.748977+0000","last_peered":"2024-05-15T13:10:11.748977+0000","last_clean":"2024-05-15T13:10:11.748977+0000","last_became_active":"2024-05-15T12:35:50.518230+0000","last_became_peered":"2024-05-15T12:35:50.518230+0000","last_unstale":"2024-05-15T13:10:11.748977+0000","last_undegraded":"2024-05-15T13:10:11.748977+0000","last_fullsized":"2024-05-15T13:10:11.748977+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:55:42.493548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.16","version":"180'4","reported_seq":2351,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.767988+0000","last_change":"2024-05-15T12:35:47.452786+0000","last_active":"2024-05-15T13:10:11.767988+0000","last_peered":"2024-05-15T13:10:11.767988+0000","last_clean":"2024-05-15T13:10:11.767988+0000","last_became_active":"2024-05-15T12:35:47.446210+0000","last_became_peered":"2024-05-15T12:35:47.446210+0000","last_unstale":"2024-05-15T13:10:11.767988+0000","last_undegraded":"2024-05-15T13:10:11.767988+0000","last_fullsized":"2024-05-15T13:10:11.767988+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:10:41.816557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.15","version":"190'2","reported_seq":2366,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.514832+0000","last_change":"2024-05-15T12:35:49.852126+0000","last_active":"2024-05-15T13:10:09.514832+0000","last_peered":"2024-05-15T13:10:09.514832+0000","last_clean":"2024-05-15T13:10:09.514832+0000","last_became_active":"2024-05-15T12:35:49.823395+0000","last_became_peered":"2024-05-15T12:35:49.823395+0000","last_unstale":"2024-05-15T13:10:09.514832+0000","last_undegraded":"2024-05-15T13:10:09.514832+0000","last_fullsized":"2024-05-15T13:10:09.514832+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:35:00.942178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.14","version":"0'0","reported_seq":2361,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.795951+0000","last_change":"2024-05-15T12:35:47.935648+0000","last_active":"2024-05-15T13:10:11.795951+0000","last_peered":"2024-05-15T13:10:11.795951+0000","last_clean":"2024-05-15T13:10:11.795951+0000","last_became_active":"2024-05-15T12:35:47.926796+0000","last_became_peered":"2024-05-15T12:35:47.926796+0000","last_unstale":"2024-05-15T13:10:11.795951+0000","last_undegraded":"2024-05-15T13:10:11.795951+0000","last_fullsized":"2024-05-15T13:10:11.795951+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:12:08.277421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.13","version":"171'1","reported_seq":2376,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.810073+0000","last_change":"2024-05-15T12:35:28.188222+0000","last_active":"2024-05-15T13:10:11.810073+0000","last_peered":"2024-05-15T13:10:11.810073+0000","last_clean":"2024-05-15T13:10:11.810073+0000","last_became_active":"2024-05-15T12:35:28.182751+0000","last_became_peered":"2024-05-15T12:35:28.182751+0000","last_unstale":"2024-05-15T13:10:11.810073+0000","last_undegraded":"2024-05-15T13:10:11.810073+0000","last_fullsized":"2024-05-15T13:10:11.810073+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:38:40.701230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.12","version":"79'1","reported_seq":2374,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.716443+0000","last_change":"2024-05-15T12:35:27.725919+0000","last_active":"2024-05-15T13:10:11.716443+0000","last_peered":"2024-05-15T13:10:11.716443+0000","last_clean":"2024-05-15T13:10:11.716443+0000","last_became_active":"2024-05-15T12:35:27.719548+0000","last_became_peered":"2024-05-15T12:35:27.719548+0000","last_unstale":"2024-05-15T13:10:11.716443+0000","last_undegraded":"2024-05-15T13:10:11.716443+0000","last_fullsized":"2024-05-15T13:10:11.716443+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:26:15.224906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.11","version":"172'1","reported_seq":2364,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.726278+0000","last_change":"2024-05-15T12:35:50.434980+0000","last_active":"2024-05-15T13:10:11.726278+0000","last_peered":"2024-05-15T13:10:11.726278+0000","last_clean":"2024-05-15T13:10:11.726278+0000","last_became_active":"2024-05-15T12:35:50.426351+0000","last_became_peered":"2024-05-15T12:35:50.426351+0000","last_unstale":"2024-05-15T13:10:11.726278+0000","last_undegraded":"2024-05-15T13:10:11.726278+0000","last_fullsized":"2024-05-15T13:10:11.726278+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:11:28.596763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.10","version":"190'3","reported_seq":2361,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.735401+0000","last_change":"2024-05-15T12:35:34.592833+0000","last_active":"2024-05-15T13:10:11.735401+0000","last_peered":"2024-05-15T13:10:11.735401+0000","last_clean":"2024-05-15T13:10:11.735401+0000","last_became_active":"2024-05-15T12:35:34.587282+0000","last_became_peered":"2024-05-15T12:35:34.587282+0000","last_unstale":"2024-05-15T13:10:11.735401+0000","last_undegraded":"2024-05-15T13:10:11.735401+0000","last_fullsized":"2024-05-15T13:10:11.735401+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:35.500859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.f","version":"155'2","reported_seq":2357,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.282099+0000","last_change":"2024-05-15T12:35:47.741135+0000","last_active":"2024-05-15T13:10:13.282099+0000","last_peered":"2024-05-15T13:10:13.282099+0000","last_clean":"2024-05-15T13:10:13.282099+0000","last_became_active":"2024-05-15T12:35:47.063782+0000","last_became_peered":"2024-05-15T12:35:47.063782+0000","last_unstale":"2024-05-15T13:10:13.282099+0000","last_undegraded":"2024-05-15T13:10:13.282099+0000","last_fullsized":"2024-05-15T13:10:13.282099+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:05:49.823797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.e","version":"174'1","reported_seq":2374,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.779790+0000","last_change":"2024-05-15T12:35:28.039423+0000","last_active":"2024-05-15T13:10:11.779790+0000","last_peered":"2024-05-15T13:10:11.779790+0000","last_clean":"2024-05-15T13:10:11.779790+0000","last_became_active":"2024-05-15T12:35:28.033694+0000","last_became_peered":"2024-05-15T12:35:28.033694+0000","last_unstale":"2024-05-15T13:10:11.779790+0000","last_undegraded":"2024-05-15T13:10:11.779790+0000","last_fullsized":"2024-05-15T13:10:11.779790+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:13:51.876959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.d","version":"175'3","reported_seq":2365,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.740170+0000","last_change":"2024-05-15T12:35:50.592101+0000","last_active":"2024-05-15T13:10:11.740170+0000","last_peered":"2024-05-15T13:10:11.740170+0000","last_clean":"2024-05-15T13:10:11.740170+0000","last_became_active":"2024-05-15T12:35:50.586234+0000","last_became_peered":"2024-05-15T12:35:50.586234+0000","last_unstale":"2024-05-15T13:10:11.740170+0000","last_undegraded":"2024-05-15T13:10:11.740170+0000","last_fullsized":"2024-05-15T13:10:11.740170+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:23:52.114251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.c","version":"190'1","reported_seq":2370,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.080944+0000","last_change":"2024-05-15T12:35:39.569380+0000","last_active":"2024-05-15T13:10:14.080944+0000","last_peered":"2024-05-15T13:10:14.080944+0000","last_clean":"2024-05-15T13:10:14.080944+0000","last_became_active":"2024-05-15T12:35:39.550911+0000","last_became_peered":"2024-05-15T12:35:39.550911+0000","last_unstale":"2024-05-15T13:10:14.080944+0000","last_undegraded":"2024-05-15T13:10:14.080944+0000","last_fullsized":"2024-05-15T13:10:14.080944+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:11:58.527351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.b","version":"186'2","reported_seq":2365,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.748051+0000","last_change":"2024-05-15T12:35:22.786550+0000","last_active":"2024-05-15T13:10:11.748051+0000","last_peered":"2024-05-15T13:10:11.748051+0000","last_clean":"2024-05-15T13:10:11.748051+0000","last_became_active":"2024-05-15T12:35:22.781126+0000","last_became_peered":"2024-05-15T12:35:22.781126+0000","last_unstale":"2024-05-15T13:10:11.748051+0000","last_undegraded":"2024-05-15T13:10:11.748051+0000","last_fullsized":"2024-05-15T13:10:11.748051+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:11:51.197065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.a","version":"170'1","reported_seq":2364,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.778652+0000","last_change":"2024-05-15T12:35:47.948588+0000","last_active":"2024-05-15T13:10:11.778652+0000","last_peered":"2024-05-15T13:10:11.778652+0000","last_clean":"2024-05-15T13:10:11.778652+0000","last_became_active":"2024-05-15T12:35:47.943168+0000","last_became_peered":"2024-05-15T12:35:47.943168+0000","last_unstale":"2024-05-15T13:10:11.778652+0000","last_undegraded":"2024-05-15T13:10:11.778652+0000","last_fullsized":"2024-05-15T13:10:11.778652+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:00:16.296438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.9","version":"190'1","reported_seq":2366,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.456092+0000","last_change":"2024-05-15T12:35:49.112002+0000","last_active":"2024-05-15T13:10:09.456092+0000","last_peered":"2024-05-15T13:10:09.456092+0000","last_clean":"2024-05-15T13:10:09.456092+0000","last_became_active":"2024-05-15T12:35:49.106669+0000","last_became_peered":"2024-05-15T12:35:49.106669+0000","last_unstale":"2024-05-15T13:10:09.456092+0000","last_undegraded":"2024-05-15T13:10:09.456092+0000","last_fullsized":"2024-05-15T13:10:09.456092+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:32:01.416983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.8","version":"0'0","reported_seq":2375,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.701935+0000","last_change":"2024-05-15T12:35:27.822672+0000","last_active":"2024-05-15T13:10:11.701935+0000","last_peered":"2024-05-15T13:10:11.701935+0000","last_clean":"2024-05-15T13:10:11.701935+0000","last_became_active":"2024-05-15T12:35:27.816334+0000","last_became_peered":"2024-05-15T12:35:27.816334+0000","last_unstale":"2024-05-15T13:10:11.701935+0000","last_undegraded":"2024-05-15T13:10:11.701935+0000","last_fullsized":"2024-05-15T13:10:11.701935+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:10:34.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.7","version":"172'1","reported_seq":2365,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.527336+0000","last_change":"2024-05-15T12:35:47.775023+0000","last_active":"2024-05-15T13:10:09.527336+0000","last_peered":"2024-05-15T13:10:09.527336+0000","last_clean":"2024-05-15T13:10:09.527336+0000","last_became_active":"2024-05-15T12:35:47.766045+0000","last_became_peered":"2024-05-15T12:35:47.766045+0000","last_unstale":"2024-05-15T13:10:09.527336+0000","last_undegraded":"2024-05-15T13:10:09.527336+0000","last_fullsized":"2024-05-15T13:10:09.527336+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:35:16.103745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.6","version":"0'0","reported_seq":2377,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.056994+0000","last_change":"2024-05-15T12:35:59.068641+0000","last_active":"2024-05-15T13:10:14.056994+0000","last_peered":"2024-05-15T13:10:14.056994+0000","last_clean":"2024-05-15T13:10:14.056994+0000","last_became_active":"2024-05-15T12:35:59.010795+0000","last_became_peered":"2024-05-15T12:35:59.010795+0000","last_unstale":"2024-05-15T13:10:14.056994+0000","last_undegraded":"2024-05-15T13:10:14.056994+0000","last_fullsized":"2024-05-15T13:10:14.056994+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:26:58.973165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.5","version":"161'1","reported_seq":2357,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.304715+0000","last_change":"2024-05-15T12:36:31.757984+0000","last_active":"2024-05-15T13:10:13.304715+0000","last_peered":"2024-05-15T13:10:13.304715+0000","last_clean":"2024-05-15T13:10:13.304715+0000","last_became_active":"2024-05-15T12:36:31.177992+0000","last_became_peered":"2024-05-15T12:36:31.177992+0000","last_unstale":"2024-05-15T13:10:13.304715+0000","last_undegraded":"2024-05-15T13:10:13.304715+0000","last_fullsized":"2024-05-15T13:10:13.304715+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:51:21.937298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.4","version":"0'0","reported_seq":2375,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.047899+0000","last_change":"2024-05-15T12:35:59.243799+0000","last_active":"2024-05-15T13:10:14.047899+0000","last_peered":"2024-05-15T13:10:14.047899+0000","last_clean":"2024-05-15T13:10:14.047899+0000","last_became_active":"2024-05-15T12:35:59.238121+0000","last_became_peered":"2024-05-15T12:35:59.238121+0000","last_unstale":"2024-05-15T13:10:14.047899+0000","last_undegraded":"2024-05-15T13:10:14.047899+0000","last_fullsized":"2024-05-15T13:10:14.047899+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:44:24.673671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.3","version":"190'4","reported_seq":2369,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.476586+0000","last_change":"2024-05-15T12:35:47.667458+0000","last_active":"2024-05-15T13:10:09.476586+0000","last_peered":"2024-05-15T13:10:09.476586+0000","last_clean":"2024-05-15T13:10:09.476586+0000","last_became_active":"2024-05-15T12:35:47.639452+0000","last_became_peered":"2024-05-15T12:35:47.639452+0000","last_unstale":"2024-05-15T13:10:09.476586+0000","last_undegraded":"2024-05-15T13:10:09.476586+0000","last_fullsized":"2024-05-15T13:10:09.476586+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:00:58.006820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":704,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.2","version":"143'1","reported_seq":2389,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.037872+0000","last_change":"2024-05-15T12:35:32.749852+0000","last_active":"2024-05-15T13:10:14.037872+0000","last_peered":"2024-05-15T13:10:14.037872+0000","last_clean":"2024-05-15T13:10:14.037872+0000","last_became_active":"2024-05-15T12:35:32.699857+0000","last_became_peered":"2024-05-15T12:35:32.699857+0000","last_unstale":"2024-05-15T13:10:14.037872+0000","last_undegraded":"2024-05-15T13:10:14.037872+0000","last_fullsized":"2024-05-15T13:10:14.037872+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:24:48.769777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1","version":"190'3","reported_seq":2360,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.298789+0000","last_change":"2024-05-15T12:36:13.620976+0000","last_active":"2024-05-15T13:10:13.298789+0000","last_peered":"2024-05-15T13:10:13.298789+0000","last_clean":"2024-05-15T13:10:13.298789+0000","last_became_active":"2024-05-15T12:36:13.614839+0000","last_became_peered":"2024-05-15T12:36:13.614839+0000","last_unstale":"2024-05-15T13:10:13.298789+0000","last_undegraded":"2024-05-15T13:10:13.298789+0000","last_fullsized":"2024-05-15T13:10:13.298789+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:01:15.571189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.0","version":"166'1","reported_seq":2352,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.741098+0000","last_change":"2024-05-15T12:35:47.277656+0000","last_active":"2024-05-15T13:10:11.741098+0000","last_peered":"2024-05-15T13:10:11.741098+0000","last_clean":"2024-05-15T13:10:11.741098+0000","last_became_active":"2024-05-15T12:35:47.271050+0000","last_became_peered":"2024-05-15T12:35:47.271050+0000","last_unstale":"2024-05-15T13:10:11.741098+0000","last_undegraded":"2024-05-15T13:10:11.741098+0000","last_fullsized":"2024-05-15T13:10:11.741098+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:12:17.443633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":2399,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.279993+0000","last_change":"2024-05-15T12:34:57.169160+0000","last_active":"2024-05-15T13:10:13.279993+0000","last_peered":"2024-05-15T13:10:13.279993+0000","last_clean":"2024-05-15T13:10:13.279993+0000","last_became_active":"2024-05-15T12:34:57.154951+0000","last_became_peered":"2024-05-15T12:34:57.154951+0000","last_unstale":"2024-05-15T13:10:13.279993+0000","last_undegraded":"2024-05-15T13:10:13.279993+0000","last_fullsized":"2024-05-15T13:10:13.279993+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:37:02.028559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":2401,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.336386+0000","last_change":"2024-05-15T12:34:54.507835+0000","last_active":"2024-05-15T13:10:13.336386+0000","last_peered":"2024-05-15T13:10:13.336386+0000","last_clean":"2024-05-15T13:10:13.336386+0000","last_became_active":"2024-05-15T12:34:54.498165+0000","last_became_peered":"2024-05-15T12:34:54.498165+0000","last_unstale":"2024-05-15T13:10:13.336386+0000","last_undegraded":"2024-05-15T13:10:13.336386+0000","last_fullsized":"2024-05-15T13:10:13.336386+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:35:01.164627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":2405,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.742356+0000","last_change":"2024-05-15T12:34:52.684394+0000","last_active":"2024-05-15T13:10:11.742356+0000","last_peered":"2024-05-15T13:10:11.742356+0000","last_clean":"2024-05-15T13:10:11.742356+0000","last_became_active":"2024-05-15T12:34:52.677394+0000","last_became_peered":"2024-05-15T12:34:52.677394+0000","last_unstale":"2024-05-15T13:10:11.742356+0000","last_undegraded":"2024-05-15T13:10:11.742356+0000","last_fullsized":"2024-05-15T13:10:11.742356+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:10:31.152782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.494523+0000","last_change":"2024-05-15T12:34:50.121401+0000","last_active":"2024-05-15T13:10:09.494523+0000","last_peered":"2024-05-15T13:10:09.494523+0000","last_clean":"2024-05-15T13:10:09.494523+0000","last_became_active":"2024-05-15T12:34:50.114215+0000","last_became_peered":"2024-05-15T12:34:50.114215+0000","last_unstale":"2024-05-15T13:10:09.494523+0000","last_undegraded":"2024-05-15T13:10:09.494523+0000","last_fullsized":"2024-05-15T13:10:09.494523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:04:16.442442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":2413,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.939553+0000","last_change":"2024-05-15T12:34:54.772141+0000","last_active":"2024-05-15T13:10:13.939553+0000","last_peered":"2024-05-15T13:10:13.939553+0000","last_clean":"2024-05-15T13:10:13.939553+0000","last_became_active":"2024-05-15T12:34:54.762972+0000","last_became_peered":"2024-05-15T12:34:54.762972+0000","last_unstale":"2024-05-15T13:10:13.939553+0000","last_undegraded":"2024-05-15T13:10:13.939553+0000","last_fullsized":"2024-05-15T13:10:13.939553+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:01:23.482313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.780442+0000","last_change":"2024-05-15T12:34:34.684239+0000","last_active":"2024-05-15T13:10:11.780442+0000","last_peered":"2024-05-15T13:10:11.780442+0000","last_clean":"2024-05-15T13:10:11.780442+0000","last_became_active":"2024-05-15T12:34:34.670411+0000","last_became_peered":"2024-05-15T12:34:34.670411+0000","last_unstale":"2024-05-15T13:10:11.780442+0000","last_undegraded":"2024-05-15T13:10:11.780442+0000","last_fullsized":"2024-05-15T13:10:11.780442+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:15:28.642471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.458171+0000","last_change":"2024-05-15T12:34:50.164567+0000","last_active":"2024-05-15T13:10:09.458171+0000","last_peered":"2024-05-15T13:10:09.458171+0000","last_clean":"2024-05-15T13:10:09.458171+0000","last_became_active":"2024-05-15T12:34:50.159068+0000","last_became_peered":"2024-05-15T12:34:50.159068+0000","last_unstale":"2024-05-15T13:10:09.458171+0000","last_undegraded":"2024-05-15T13:10:09.458171+0000","last_fullsized":"2024-05-15T13:10:09.458171+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:05:11.053203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":2414,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.449508+0000","last_change":"2024-05-15T12:34:39.918992+0000","last_active":"2024-05-15T13:10:09.449508+0000","last_peered":"2024-05-15T13:10:09.449508+0000","last_clean":"2024-05-15T13:10:09.449508+0000","last_became_active":"2024-05-15T12:34:39.907050+0000","last_became_peered":"2024-05-15T12:34:39.907050+0000","last_unstale":"2024-05-15T13:10:09.449508+0000","last_undegraded":"2024-05-15T13:10:09.449508+0000","last_fullsized":"2024-05-15T13:10:09.449508+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:10:12.279986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":2392,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.796844+0000","last_change":"2024-05-15T12:34:44.459281+0000","last_active":"2024-05-15T13:10:11.796844+0000","last_peered":"2024-05-15T13:10:11.796844+0000","last_clean":"2024-05-15T13:10:11.796844+0000","last_became_active":"2024-05-15T12:34:44.443445+0000","last_became_peered":"2024-05-15T12:34:44.443445+0000","last_unstale":"2024-05-15T13:10:11.796844+0000","last_undegraded":"2024-05-15T13:10:11.796844+0000","last_fullsized":"2024-05-15T13:10:11.796844+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:03:29.028978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":2394,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.771857+0000","last_change":"2024-05-15T12:34:38.200148+0000","last_active":"2024-05-15T13:10:11.771857+0000","last_peered":"2024-05-15T13:10:11.771857+0000","last_clean":"2024-05-15T13:10:11.771857+0000","last_became_active":"2024-05-15T12:34:38.194477+0000","last_became_peered":"2024-05-15T12:34:38.194477+0000","last_unstale":"2024-05-15T13:10:11.771857+0000","last_undegraded":"2024-05-15T13:10:11.771857+0000","last_fullsized":"2024-05-15T13:10:11.771857+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:28:19.294534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":2400,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.306918+0000","last_change":"2024-05-15T12:34:42.333270+0000","last_active":"2024-05-15T13:10:13.306918+0000","last_peered":"2024-05-15T13:10:13.306918+0000","last_clean":"2024-05-15T13:10:13.306918+0000","last_became_active":"2024-05-15T12:34:42.317054+0000","last_became_peered":"2024-05-15T12:34:42.317054+0000","last_unstale":"2024-05-15T13:10:13.306918+0000","last_undegraded":"2024-05-15T13:10:13.306918+0000","last_fullsized":"2024-05-15T13:10:13.306918+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:15:51.004019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":2411,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.078980+0000","last_change":"2024-05-15T12:34:48.752104+0000","last_active":"2024-05-15T13:10:14.078980+0000","last_peered":"2024-05-15T13:10:14.078980+0000","last_clean":"2024-05-15T13:10:14.078980+0000","last_became_active":"2024-05-15T12:34:48.742068+0000","last_became_peered":"2024-05-15T12:34:48.742068+0000","last_unstale":"2024-05-15T13:10:14.078980+0000","last_undegraded":"2024-05-15T13:10:14.078980+0000","last_fullsized":"2024-05-15T13:10:14.078980+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:18:38.517375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"26'76","reported_seq":2461,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.243120+0000","last_change":"2024-05-15T12:32:43.562093+0000","last_active":"2024-05-15T13:10:13.243120+0000","last_peered":"2024-05-15T13:10:13.243120+0000","last_clean":"2024-05-15T13:10:13.243120+0000","last_became_active":"2024-05-15T12:32:42.889644+0000","last_became_peered":"2024-05-15T12:32:42.889644+0000","last_unstale":"2024-05-15T13:10:13.243120+0000","last_undegraded":"2024-05-15T13:10:13.243120+0000","last_fullsized":"2024-05-15T13:10:13.243120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:30.812143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:30.812143+0000","last_clean_scrub_stamp":"2024-05-15T12:32:30.812143+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:45:22.988060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2420,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.718656+0000","last_change":"2024-05-15T12:32:47.722217+0000","last_active":"2024-05-15T13:10:11.718656+0000","last_peered":"2024-05-15T13:10:11.718656+0000","last_clean":"2024-05-15T13:10:11.718656+0000","last_became_active":"2024-05-15T12:32:47.716674+0000","last_became_peered":"2024-05-15T12:32:47.716674+0000","last_unstale":"2024-05-15T13:10:11.718656+0000","last_undegraded":"2024-05-15T13:10:11.718656+0000","last_fullsized":"2024-05-15T13:10:11.718656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:25:03.820946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":2413,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.530936+0000","last_change":"2024-05-15T12:34:42.947741+0000","last_active":"2024-05-15T13:10:09.530936+0000","last_peered":"2024-05-15T13:10:09.530936+0000","last_clean":"2024-05-15T13:10:09.530936+0000","last_became_active":"2024-05-15T12:34:42.942214+0000","last_became_peered":"2024-05-15T12:34:42.942214+0000","last_unstale":"2024-05-15T13:10:09.530936+0000","last_undegraded":"2024-05-15T13:10:09.530936+0000","last_fullsized":"2024-05-15T13:10:09.530936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:08:22.005787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2421,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.702999+0000","last_change":"2024-05-15T12:32:47.459103+0000","last_active":"2024-05-15T13:10:11.702999+0000","last_peered":"2024-05-15T13:10:11.702999+0000","last_clean":"2024-05-15T13:10:11.702999+0000","last_became_active":"2024-05-15T12:32:47.430194+0000","last_became_peered":"2024-05-15T12:32:47.430194+0000","last_unstale":"2024-05-15T13:10:11.702999+0000","last_undegraded":"2024-05-15T13:10:11.702999+0000","last_fullsized":"2024-05-15T13:10:11.702999+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:14:58.294854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.744310+0000","last_change":"2024-05-15T12:34:52.363872+0000","last_active":"2024-05-15T13:10:11.744310+0000","last_peered":"2024-05-15T13:10:11.744310+0000","last_clean":"2024-05-15T13:10:11.744310+0000","last_became_active":"2024-05-15T12:34:52.357328+0000","last_became_peered":"2024-05-15T12:34:52.357328+0000","last_unstale":"2024-05-15T13:10:11.744310+0000","last_undegraded":"2024-05-15T13:10:11.744310+0000","last_fullsized":"2024-05-15T13:10:11.744310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:05:48.456769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2408,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.743300+0000","last_change":"2024-05-15T12:32:47.463044+0000","last_active":"2024-05-15T13:10:11.743300+0000","last_peered":"2024-05-15T13:10:11.743300+0000","last_clean":"2024-05-15T13:10:11.743300+0000","last_became_active":"2024-05-15T12:32:47.432679+0000","last_became_peered":"2024-05-15T12:32:47.432679+0000","last_unstale":"2024-05-15T13:10:11.743300+0000","last_undegraded":"2024-05-15T13:10:11.743300+0000","last_fullsized":"2024-05-15T13:10:11.743300+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:13:56.912246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":2414,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.800112+0000","last_change":"2024-05-15T12:34:36.929836+0000","last_active":"2024-05-15T13:10:11.800112+0000","last_peered":"2024-05-15T13:10:11.800112+0000","last_clean":"2024-05-15T13:10:11.800112+0000","last_became_active":"2024-05-15T12:34:36.924450+0000","last_became_peered":"2024-05-15T12:34:36.924450+0000","last_unstale":"2024-05-15T13:10:11.800112+0000","last_undegraded":"2024-05-15T13:10:11.800112+0000","last_fullsized":"2024-05-15T13:10:11.800112+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:13:01.098128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2423,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.712047+0000","last_change":"2024-05-15T12:36:58.748207+0000","last_active":"2024-05-15T13:10:11.712047+0000","last_peered":"2024-05-15T13:10:11.712047+0000","last_clean":"2024-05-15T13:10:11.712047+0000","last_became_active":"2024-05-15T12:32:47.670226+0000","last_became_peered":"2024-05-15T12:32:47.670226+0000","last_unstale":"2024-05-15T13:10:11.712047+0000","last_undegraded":"2024-05-15T13:10:11.712047+0000","last_fullsized":"2024-05-15T13:10:11.712047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:41:09.623434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.751327+0000","last_change":"2024-05-15T12:34:51.754656+0000","last_active":"2024-05-15T13:10:11.751327+0000","last_peered":"2024-05-15T13:10:11.751327+0000","last_clean":"2024-05-15T13:10:11.751327+0000","last_became_active":"2024-05-15T12:34:51.746853+0000","last_became_peered":"2024-05-15T12:34:51.746853+0000","last_unstale":"2024-05-15T13:10:11.751327+0000","last_undegraded":"2024-05-15T13:10:11.751327+0000","last_fullsized":"2024-05-15T13:10:11.751327+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:59:19.605051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2426,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.497551+0000","last_change":"2024-05-15T12:32:47.687489+0000","last_active":"2024-05-15T13:10:09.497551+0000","last_peered":"2024-05-15T13:10:09.497551+0000","last_clean":"2024-05-15T13:10:09.497551+0000","last_became_active":"2024-05-15T12:32:47.680498+0000","last_became_peered":"2024-05-15T12:32:47.680498+0000","last_unstale":"2024-05-15T13:10:09.497551+0000","last_undegraded":"2024-05-15T13:10:09.497551+0000","last_fullsized":"2024-05-15T13:10:09.497551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:09:36.650810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"39'7","reported_seq":2427,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.048781+0000","last_change":"2024-05-15T12:34:50.381955+0000","last_active":"2024-05-15T13:10:14.048781+0000","last_peered":"2024-05-15T13:10:14.048781+0000","last_clean":"2024-05-15T13:10:14.048781+0000","last_became_active":"2024-05-15T12:34:50.371773+0000","last_became_peered":"2024-05-15T12:34:50.371773+0000","last_unstale":"2024-05-15T13:10:14.048781+0000","last_undegraded":"2024-05-15T13:10:14.048781+0000","last_fullsized":"2024-05-15T13:10:14.048781+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:40:48.381654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2419,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.746265+0000","last_change":"2024-05-15T12:32:47.531833+0000","last_active":"2024-05-15T13:10:11.746265+0000","last_peered":"2024-05-15T13:10:11.746265+0000","last_clean":"2024-05-15T13:10:11.746265+0000","last_became_active":"2024-05-15T12:32:47.526275+0000","last_became_peered":"2024-05-15T12:32:47.526275+0000","last_unstale":"2024-05-15T13:10:11.746265+0000","last_undegraded":"2024-05-15T13:10:11.746265+0000","last_fullsized":"2024-05-15T13:10:11.746265+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:32:09.483290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":2404,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.241050+0000","last_change":"2024-05-15T12:34:38.895220+0000","last_active":"2024-05-15T13:10:13.241050+0000","last_peered":"2024-05-15T13:10:13.241050+0000","last_clean":"2024-05-15T13:10:13.241050+0000","last_became_active":"2024-05-15T12:34:38.888318+0000","last_became_peered":"2024-05-15T12:34:38.888318+0000","last_unstale":"2024-05-15T13:10:13.241050+0000","last_undegraded":"2024-05-15T13:10:13.241050+0000","last_fullsized":"2024-05-15T13:10:13.241050+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:38:34.358044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2424,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.818412+0000","last_change":"2024-05-15T12:32:48.309674+0000","last_active":"2024-05-15T13:10:11.818412+0000","last_peered":"2024-05-15T13:10:11.818412+0000","last_clean":"2024-05-15T13:10:11.818412+0000","last_became_active":"2024-05-15T12:32:48.283635+0000","last_became_peered":"2024-05-15T12:32:48.283635+0000","last_unstale":"2024-05-15T13:10:11.818412+0000","last_undegraded":"2024-05-15T13:10:11.818412+0000","last_fullsized":"2024-05-15T13:10:11.818412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:07:32.002393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":2414,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.480560+0000","last_change":"2024-05-15T12:34:48.672942+0000","last_active":"2024-05-15T13:10:09.480560+0000","last_peered":"2024-05-15T13:10:09.480560+0000","last_clean":"2024-05-15T13:10:09.480560+0000","last_became_active":"2024-05-15T12:34:48.664879+0000","last_became_peered":"2024-05-15T12:34:48.664879+0000","last_unstale":"2024-05-15T13:10:09.480560+0000","last_undegraded":"2024-05-15T13:10:09.480560+0000","last_fullsized":"2024-05-15T13:10:09.480560+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:48:54.955944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2418,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.753278+0000","last_change":"2024-05-15T12:32:48.258451+0000","last_active":"2024-05-15T13:10:11.753278+0000","last_peered":"2024-05-15T13:10:11.753278+0000","last_clean":"2024-05-15T13:10:11.753278+0000","last_became_active":"2024-05-15T12:32:48.252725+0000","last_became_peered":"2024-05-15T12:32:48.252725+0000","last_unstale":"2024-05-15T13:10:11.753278+0000","last_undegraded":"2024-05-15T13:10:11.753278+0000","last_fullsized":"2024-05-15T13:10:11.753278+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:01:52.775805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":2414,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.782737+0000","last_change":"2024-05-15T12:34:34.676904+0000","last_active":"2024-05-15T13:10:11.782737+0000","last_peered":"2024-05-15T13:10:11.782737+0000","last_clean":"2024-05-15T13:10:11.782737+0000","last_became_active":"2024-05-15T12:34:34.660678+0000","last_became_peered":"2024-05-15T12:34:34.660678+0000","last_unstale":"2024-05-15T13:10:11.782737+0000","last_undegraded":"2024-05-15T13:10:11.782737+0000","last_fullsized":"2024-05-15T13:10:11.782737+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:39:07.124084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":2394,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.739089+0000","last_change":"2024-05-15T12:34:44.299228+0000","last_active":"2024-05-15T13:10:11.739089+0000","last_peered":"2024-05-15T13:10:11.739089+0000","last_clean":"2024-05-15T13:10:11.739089+0000","last_became_active":"2024-05-15T12:34:44.277750+0000","last_became_peered":"2024-05-15T12:34:44.277750+0000","last_unstale":"2024-05-15T13:10:11.739089+0000","last_undegraded":"2024-05-15T13:10:11.739089+0000","last_fullsized":"2024-05-15T13:10:11.739089+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:00:02.340970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":2412,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.046745+0000","last_change":"2024-05-15T12:34:39.506099+0000","last_active":"2024-05-15T13:10:14.046745+0000","last_peered":"2024-05-15T13:10:14.046745+0000","last_clean":"2024-05-15T13:10:14.046745+0000","last_became_active":"2024-05-15T12:34:39.369183+0000","last_became_peered":"2024-05-15T12:34:39.369183+0000","last_unstale":"2024-05-15T13:10:14.046745+0000","last_undegraded":"2024-05-15T13:10:14.046745+0000","last_fullsized":"2024-05-15T13:10:14.046745+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:41:03.408587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":2402,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.286633+0000","last_change":"2024-05-15T12:34:38.831156+0000","last_active":"2024-05-15T13:10:13.286633+0000","last_peered":"2024-05-15T13:10:13.286633+0000","last_clean":"2024-05-15T13:10:13.286633+0000","last_became_active":"2024-05-15T12:34:38.812314+0000","last_became_peered":"2024-05-15T12:34:38.812314+0000","last_unstale":"2024-05-15T13:10:13.286633+0000","last_undegraded":"2024-05-15T13:10:13.286633+0000","last_fullsized":"2024-05-15T13:10:13.286633+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:45:31.161837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":2393,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.745771+0000","last_change":"2024-05-15T12:34:36.784260+0000","last_active":"2024-05-15T13:10:11.745771+0000","last_peered":"2024-05-15T13:10:11.745771+0000","last_clean":"2024-05-15T13:10:11.745771+0000","last_became_active":"2024-05-15T12:34:36.777340+0000","last_became_peered":"2024-05-15T12:34:36.777340+0000","last_unstale":"2024-05-15T13:10:11.745771+0000","last_undegraded":"2024-05-15T13:10:11.745771+0000","last_fullsized":"2024-05-15T13:10:11.745771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:27:38.620258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":2410,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.077032+0000","last_change":"2024-05-15T12:34:51.056513+0000","last_active":"2024-05-15T13:10:14.077032+0000","last_peered":"2024-05-15T13:10:14.077032+0000","last_clean":"2024-05-15T13:10:14.077032+0000","last_became_active":"2024-05-15T12:34:51.040048+0000","last_became_peered":"2024-05-15T12:34:51.040048+0000","last_unstale":"2024-05-15T13:10:14.077032+0000","last_undegraded":"2024-05-15T13:10:14.077032+0000","last_fullsized":"2024-05-15T13:10:14.077032+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:32:15.066105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":2412,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-05-15T13:10:09.516901+0000","last_change":"2024-05-15T12:34:51.656110+0000","last_active":"2024-05-15T13:10:09.516901+0000","last_peered":"2024-05-15T13:10:09.516901+0000","last_clean":"2024-05-15T13:10:09.516901+0000","last_became_active":"2024-05-15T12:34:51.649910+0000","last_became_peered":"2024-05-15T12:34:51.649910+0000","last_unstale":"2024-05-15T13:10:09.516901+0000","last_undegraded":"2024-05-15T13:10:09.516901+0000","last_fullsized":"2024-05-15T13:10:09.516901+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:01:47.425779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":2396,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.770003+0000","last_change":"2024-05-15T12:34:44.408851+0000","last_active":"2024-05-15T13:10:11.770003+0000","last_peered":"2024-05-15T13:10:11.770003+0000","last_clean":"2024-05-15T13:10:11.770003+0000","last_became_active":"2024-05-15T12:34:44.399991+0000","last_became_peered":"2024-05-15T12:34:44.399991+0000","last_unstale":"2024-05-15T13:10:11.770003+0000","last_undegraded":"2024-05-15T13:10:11.770003+0000","last_fullsized":"2024-05-15T13:10:11.770003+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:40:47.114950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":2424,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.064927+0000","last_change":"2024-05-15T12:34:38.557851+0000","last_active":"2024-05-15T13:10:14.064927+0000","last_peered":"2024-05-15T13:10:14.064927+0000","last_clean":"2024-05-15T13:10:14.064927+0000","last_became_active":"2024-05-15T12:34:38.549385+0000","last_became_peered":"2024-05-15T12:34:38.549385+0000","last_unstale":"2024-05-15T13:10:14.064927+0000","last_undegraded":"2024-05-15T13:10:14.064927+0000","last_fullsized":"2024-05-15T13:10:14.064927+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:31:24.444568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":2411,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.756882+0000","last_change":"2024-05-15T12:34:36.889179+0000","last_active":"2024-05-15T13:10:11.756882+0000","last_peered":"2024-05-15T13:10:11.756882+0000","last_clean":"2024-05-15T13:10:11.756882+0000","last_became_active":"2024-05-15T12:34:36.875596+0000","last_became_peered":"2024-05-15T12:34:36.875596+0000","last_unstale":"2024-05-15T13:10:11.756882+0000","last_undegraded":"2024-05-15T13:10:11.756882+0000","last_fullsized":"2024-05-15T13:10:11.756882+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:30:10.134778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.723668+0000","last_change":"2024-05-15T12:34:51.659524+0000","last_active":"2024-05-15T13:10:11.723668+0000","last_peered":"2024-05-15T13:10:11.723668+0000","last_clean":"2024-05-15T13:10:11.723668+0000","last_became_active":"2024-05-15T12:34:51.653558+0000","last_became_peered":"2024-05-15T12:34:51.653558+0000","last_unstale":"2024-05-15T13:10:11.723668+0000","last_undegraded":"2024-05-15T13:10:11.723668+0000","last_fullsized":"2024-05-15T13:10:11.723668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:43:01.117943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.710084+0000","last_change":"2024-05-15T12:34:36.637973+0000","last_active":"2024-05-15T13:10:11.710084+0000","last_peered":"2024-05-15T13:10:11.710084+0000","last_clean":"2024-05-15T13:10:11.710084+0000","last_became_active":"2024-05-15T12:34:36.629372+0000","last_became_peered":"2024-05-15T12:34:36.629372+0000","last_unstale":"2024-05-15T13:10:11.710084+0000","last_undegraded":"2024-05-15T13:10:11.710084+0000","last_fullsized":"2024-05-15T13:10:11.710084+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:10:32.079108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":2408,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.812762+0000","last_change":"2024-05-15T12:34:49.358758+0000","last_active":"2024-05-15T13:10:11.812762+0000","last_peered":"2024-05-15T13:10:11.812762+0000","last_clean":"2024-05-15T13:10:11.812762+0000","last_became_active":"2024-05-15T12:34:49.324761+0000","last_became_peered":"2024-05-15T12:34:49.324761+0000","last_unstale":"2024-05-15T13:10:11.812762+0000","last_undegraded":"2024-05-15T13:10:11.812762+0000","last_fullsized":"2024-05-15T13:10:11.812762+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:13:55.863763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":6703,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":13406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313668,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68824,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303016,"statfs":{"total":96636764160,"available":96566288384,"internally_reserved":0,"allocated":5840896,"data_stored":3135087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]},{"osd":7,"up_from":19,"seq":81604379073,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72528,"kb_used_data":5252,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299312,"statfs":{"total":96636764160,"available":96562495488,"internally_reserved":0,"allocated":5378048,"data_stored":2731941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":19,"seq":81604379079,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72916,"kb_used_data":5636,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298924,"statfs":{"total":96636764160,"available":96562098176,"internally_reserved":0,"allocated":5771264,"data_stored":3110480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444485,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76712,"kb_used_data":5272,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295128,"statfs":{"total":96636764160,"available":96558211072,"internally_reserved":0,"allocated":5398528,"data_stored":2735851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444483,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72528,"kb_used_data":5248,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299312,"statfs":{"total":96636764160,"available":96562495488,"internally_reserved":0,"allocated":5373952,"data_stored":2742021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542599,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72580,"kb_used_data":5304,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299260,"statfs":{"total":96636764160,"available":96562442240,"internally_reserved":0,"allocated":5431296,"data_stored":2744265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509889,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68360,"kb_used_data":5244,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303480,"statfs":{"total":96636764160,"available":96566763520,"internally_reserved":0,"allocated":5369856,"data_stored":2720297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575301,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72528,"kb_used_data":5252,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299312,"statfs":{"total":96636764160,"available":96562495488,"internally_reserved":0,"allocated":5378048,"data_stored":2727897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:10:16.871 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-15T13:10:16.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T13:10:17.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.020+0000 7f83a3d28640 10 monclient: get_monmap_and_config 2024-05-15T13:10:17.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.020+0000 7f83a3d28640 10 monclient: build_initial_monmap 2024-05-15T13:10:17.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.021+0000 7f83a3d28640 10 monclient: monmap: 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:17.021520+0000 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:17.021520+0000 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:17.011 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.021+0000 7f83a3d28640 1 Processor -- start 2024-05-15T13:10:17.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.021+0000 7f83a3d28640 1 -- start start 2024-05-15T13:10:17.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.021+0000 7f83a3d28640 10 monclient: init 2024-05-15T13:10:17.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:17.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-05-15T13:10:17.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f839c0b5350 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:17.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f839c140000 0x7f839c0a9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 10 monclient(hunting): picked mon.noname-a con 0x7f839c140000 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c145b60 0x7f839c145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 10 monclient(hunting): picked mon.noname-b con 0x7f839c145b60 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c146530 con 0x7f839c0b5350 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c0b5730 con 0x7f839c140000 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0ab120 con 0x7f839c145b60 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.022+0000 7f83a3d28640 10 monclient(hunting): authenticate will time out at 3775.183926s 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.023+0000 7f83a22c5640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c145b60 0x7f839c145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.023+0000 7f83a1ac4640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f839c0b5350 0x7f839c13f8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:55228/0 (socket says 172.21.15.49:55228) 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.023+0000 7f83a1ac4640 1 -- 172.21.15.49:0/1287121600 learned_addr learned my addr 172.21.15.49:0/1287121600 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.023+0000 7f83a22c5640 10 monclient(hunting): get_auth_request con 0x7f839c145b60 auth_method 0 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.023+0000 7f83a22c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.023+0000 7f83a22c5640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.023+0000 7f83a22c5640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.023+0000 7f83a12c3640 1 --2- 172.21.15.49:0/1287121600 >> v2:172.21.15.49:3300/0 conn(0x7f839c140000 0x7f839c0a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.024+0000 7f83a22c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.024+0000 7f83a22c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.024+0000 7f83a22c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.024+0000 7f83a12c3640 10 monclient(hunting): get_auth_request con 0x7f839c140000 auth_method 0 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.024+0000 7f83a12c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.024+0000 7f83a12c3640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:17.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.024+0000 7f83a12c3640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:17.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.025+0000 7f83a22c5640 10 monclient(hunting): handle_auth_done global_id 17597 payload 274 2024-05-15T13:10:17.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.025+0000 7f83a22c5640 1 -- 172.21.15.49:0/1287121600 >> v2:172.21.15.49:3300/0 conn(0x7f839c140000 msgr2=0x7f839c0a9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:17.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.025+0000 7f83a22c5640 1 --2- 172.21.15.49:0/1287121600 >> v2:172.21.15.49:3300/0 conn(0x7f839c140000 0x7f839c0a9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.025+0000 7f83a22c5640 1 -- 172.21.15.49:0/1287121600 >> v1:172.21.15.49:6789/0 conn(0x7f839c0b5350 legacy=0x7f839c13f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:17.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.025+0000 7f83a22c5640 10 monclient: _finish_hunting 0 2024-05-15T13:10:17.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.025+0000 7f83a22c5640 1 monclient: found mon.noname-b 2024-05-15T13:10:17.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.025+0000 7f83a22c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:17.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.025+0000 7f83a22c5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T13:10:17.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.025+0000 7f83a22c5640 1 -- 172.21.15.49:0/1287121600 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c0ab400 con 0x7f839c145b60 2024-05-15T13:10:17.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.027+0000 7f83a22c5640 1 --2- 172.21.15.49:0/1287121600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c145b60 0x7f839c145ff0 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f8394009d90 tx=0x7f839402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=472f54d3cc180f40 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:17.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 1 -- 172.21.15.49:0/1287121600 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394039070 con 0x7f839c145b60 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 10 monclient: dump: 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 10 monclient: _finish_auth 0 2024-05-15T13:10:17.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 10 monclient: _check_auth_tickets 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 1 -- 172.21.15.49:0/1287121600 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839402cc50 con 0x7f839c145b60 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a3d28640 5 monclient: authenticate success, global_id 17597 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a3d28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a3d28640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 1 -- 172.21.15.49:0/1287121600 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394034440 con 0x7f839c145b60 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a3d28640 4 set_mon_vals no callback set 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.031+0000 7f83a0ac2640 10 monclient: dump: 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:17.022 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:17.023 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.023 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.023 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:17.023 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.023 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:17.023 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:17.023 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:17.023 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.032+0000 7f83a3d28640 10 monclient: shutdown 2024-05-15T13:10:17.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.032+0000 7f83a3d28640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:17.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.032+0000 7f83a3d28640 1 -- 172.21.15.49:0/1287121600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c145b60 msgr2=0x7f839c145ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.032+0000 7f83a3d28640 1 --2- 172.21.15.49:0/1287121600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c145b60 0x7f839c145ff0 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f8394009d90 tx=0x7f839402bea0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.032+0000 7f83a3d28640 1 -- 172.21.15.49:0/1287121600 shutdown_connections 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.032+0000 7f83a3d28640 1 --2- 172.21.15.49:0/1287121600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c145b60 0x7f839c145ff0 unknown :-1 s=CLOSED pgs=3118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.032+0000 7f83a3d28640 1 --2- 172.21.15.49:0/1287121600 >> v2:172.21.15.49:3300/0 conn(0x7f839c140000 0x7f839c0a9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.032+0000 7f83a3d28640 1 -- 172.21.15.49:0/1287121600 >> 172.21.15.49:0/1287121600 conn(0x7f839c066ed0 msgr2=0x7f839c0672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.032+0000 7f83a3d28640 1 -- 172.21.15.49:0/1287121600 shutdown_connections 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.032+0000 7f83a3d28640 1 -- 172.21.15.49:0/1287121600 wait complete. 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.033+0000 7f83a3d28640 10 monclient: build_initial_monmap 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.033+0000 7f83a3d28640 10 monclient: monmap: 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:17.027 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.033+0000 7f83a3d28640 1 librados: starting msgr at 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.033+0000 7f83a3d28640 1 librados: starting objecter 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.033+0000 7f83a3d28640 1 Processor -- start 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.033+0000 7f83a3d28640 1 -- start start 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.033+0000 7f83a3d28640 1 librados: setting wanted keys 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.033+0000 7f83a3d28640 1 librados: calling monclient init 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.033+0000 7f83a3d28640 10 monclient: init 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T13:10:17.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f839c0b5350 0x7f839c1db550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 10 monclient(hunting): picked mon.noname-b con 0x7f839c0b5350 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c140000 0x7f839c1378e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 10 monclient(hunting): picked mon.noname-c con 0x7f839c140000 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f839c145b60 0x7f839c1dbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 10 monclient(hunting): picked mon.noname-a con 0x7f839c145b60 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0aa470 con 0x7f839c145b60 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c14ceb0 con 0x7f839c0b5350 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0b7c10 con 0x7f839c140000 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a3d28640 10 monclient(hunting): authenticate will time out at 3775.195640s 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a22c5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f839c145b60 0x7f839c1dbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.034+0000 7f83a12c3640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c140000 0x7f839c1378e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a1ac4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f839c0b5350 0x7f839c1db550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a22c5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f839c145b60 0x7f839c1dbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59876/0 (socket says 172.21.15.49:59876) 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a22c5640 1 -- 172.21.15.49:0/3256833671 learned_addr learned my addr 172.21.15.49:0/3256833671 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a22c5640 10 monclient(hunting): get_auth_request con 0x7f839c145b60 auth_method 0 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a22c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a22c5640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a22c5640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a12c3640 10 monclient(hunting): get_auth_request con 0x7f839c140000 auth_method 0 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a12c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a12c3640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a12c3640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a1ac4640 10 monclient(hunting): get_auth_request con 0x7f839c0b5350 auth_method 0 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a1ac4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a1ac4640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:17.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a1ac4640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:17.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a22c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:17.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a22c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:17.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a22c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:17.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a1ac4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:17.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a1ac4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:17.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a1ac4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:17.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a12c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:17.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a12c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:17.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.035+0000 7f83a12c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:17.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.038+0000 7f83a1ac4640 10 monclient(hunting): handle_auth_done global_id 21994 payload 824 2024-05-15T13:10:17.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.038+0000 7f83a1ac4640 1 -- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c140000 msgr2=0x7f839c1378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:17.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.038+0000 7f83a1ac4640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c140000 0x7f839c1378e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.038+0000 7f83a1ac4640 1 -- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f839c145b60 msgr2=0x7f839c1dbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:17.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.038+0000 7f83a1ac4640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f839c145b60 0x7f839c1dbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.038+0000 7f83a1ac4640 10 monclient: _finish_hunting 0 2024-05-15T13:10:17.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.038+0000 7f83a1ac4640 1 monclient: found mon.noname-b 2024-05-15T13:10:17.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.038+0000 7f83a1ac4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:17.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.038+0000 7f83a1ac4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:17.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.038+0000 7f83a1ac4640 1 -- 172.21.15.49:0/3256833671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c0b7e80 con 0x7f839c0b5350 2024-05-15T13:10:17.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.041+0000 7f83a1ac4640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f839c0b5350 0x7f839c1db550 secure :-1 s=READY pgs=5444 cs=0 l=1 rev1=1 crypto rx=0x7f8398000f60 tx=0x7f839800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:17.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.041+0000 7f83927fc640 10 client.?.objecter ms_handle_connect 0x7f839c0b5350 2024-05-15T13:10:17.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.041+0000 7f83927fc640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:17.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 1 -- 172.21.15.49:0/3256833671 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398017070 con 0x7f839c0b5350 2024-05-15T13:10:17.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:17.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:17.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 10 monclient: dump: 2024-05-15T13:10:17.034 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 10 monclient: _finish_auth 0 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 10 monclient: _check_auth_tickets 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 1 -- 172.21.15.49:0/3256833671 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8398005ce0 con 0x7f839c0b5350 2024-05-15T13:10:17.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 1 -- 172.21.15.49:0/3256833671 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398007310 con 0x7f839c0b5350 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83927fc640 10 monclient: dump: 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83a3d28640 5 monclient: authenticate success, global_id 21994 2024-05-15T13:10:17.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83a3d28640 10 monclient: _renew_subs 2024-05-15T13:10:17.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83a3d28640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:17.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83a3d28640 1 -- 172.21.15.49:0/3256833671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839c0b8160 con 0x7f839c0b5350 2024-05-15T13:10:17.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f8393fff640 4 set_mon_vals no callback set 2024-05-15T13:10:17.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83a3d28640 10 client.21994.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:17.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83a3d28640 10 monclient: _renew_subs 2024-05-15T13:10:17.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83a3d28640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:17.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83a3d28640 1 -- 172.21.15.49:0/3256833671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c1dc080 con 0x7f839c0b5350 2024-05-15T13:10:17.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.044+0000 7f83a3d28640 1 librados: init done 2024-05-15T13:10:17.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.045+0000 7f83a3d28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:17.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.045+0000 7f83a3d28640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:17.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.045+0000 7f83a3d28640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:17.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.045+0000 7f83a3d28640 1 -- 172.21.15.49:0/3256833671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c0b6e80 con 0x7f839c0b5350 2024-05-15T13:10:17.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.298+0000 7f83927fc640 1 -- 172.21.15.49:0/3256833671 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f839801a070 con 0x7f839c0b5350 2024-05-15T13:10:17.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.298+0000 7f83927fc640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f838006cee0 0x7f838006f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.298+0000 7f83a12c3640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f838006cee0 0x7f838006f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.298+0000 7f83a12c3640 10 monclient: get_auth_request con 0x7f838006cee0 auth_method 0 2024-05-15T13:10:17.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.299+0000 7f83a12c3640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f838006cee0 0x7f838006f3a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f839c1dc3e0 tx=0x7f838c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:17.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.299+0000 7f83927fc640 10 client.21994.objecter ms_handle_connect 0x7f838006cee0 2024-05-15T13:10:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.302+0000 7f83927fc640 1 -- 172.21.15.49:0/3256833671 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1250..1250 src has 936..1250) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f83980561d0 con 0x7f839c0b5350 2024-05-15T13:10:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.302+0000 7f83927fc640 10 client.21994.objecter ms_dispatch 0x7f839c000b90 osd_map(1250..1250 src has 936..1250) v4 2024-05-15T13:10:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.302+0000 7f83927fc640 3 client.21994.objecter handle_osd_map got epochs [1250,1250] > 0 2024-05-15T13:10:17.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.302+0000 7f83927fc640 3 client.21994.objecter handle_osd_map decoding full epoch 1250 2024-05-15T13:10:17.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.302+0000 7f83927fc640 20 client.21994.objecter dump_active .. 0 homeless 2024-05-15T13:10:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.312+0000 7f83927fc640 1 -- 172.21.15.49:0/3256833671 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8398021020 con 0x7f839c0b5350 2024-05-15T13:10:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.312+0000 7f83927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.312+0000 7f83927fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.500+0000 7f83a3d28640 1 -- 172.21.15.49:0/3256833671 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f839c0b8760 con 0x7f838006cee0 2024-05-15T13:10:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.505+0000 7f83927fc640 1 -- 172.21.15.49:0/3256833671 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+197035 (secure 0 0 0) 0x7f839c0b8760 con 0x7f838006cee0 2024-05-15T13:10:17.497 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T13:10:17.498 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-05-15T13:10:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 10 librados: watch_flush enter 2024-05-15T13:10:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 10 librados: watch_flush exit 2024-05-15T13:10:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 20 client.21994.objecter shutdown clearing up homeless session... 2024-05-15T13:10:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 10 client.21994.objecter successfully canceled tick 2024-05-15T13:10:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 -- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f838006cee0 msgr2=0x7f838006f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f838006cee0 0x7f838006f3a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f839c1dc3e0 tx=0x7f838c008040 comp rx=0 tx=0).stop 2024-05-15T13:10:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 10 monclient: shutdown 2024-05-15T13:10:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 -- 172.21.15.49:0/3256833671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f839c0b5350 msgr2=0x7f839c1db550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f839c0b5350 0x7f839c1db550 secure :-1 s=READY pgs=5444 cs=0 l=1 rev1=1 crypto rx=0x7f8398000f60 tx=0x7f839800c800 comp rx=0 tx=0).stop 2024-05-15T13:10:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 -- 172.21.15.49:0/3256833671 shutdown_connections 2024-05-15T13:10:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f839c145b60 0x7f839c1dbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f839c140000 0x7f839c1378e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f838006cee0 0x7f838006f3a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 --2- 172.21.15.49:0/3256833671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f839c0b5350 0x7f839c1db550 unknown :-1 s=CLOSED pgs=5444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 -- 172.21.15.49:0/3256833671 >> 172.21.15.49:0/3256833671 conn(0x7f839c066ed0 msgr2=0x7f839c13bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 -- 172.21.15.49:0/3256833671 shutdown_connections 2024-05-15T13:10:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 -- 172.21.15.49:0/3256833671 wait complete. 2024-05-15T13:10:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.508+0000 7f83a3d28640 1 librados: shutdown 2024-05-15T13:10:17.509 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2388,"stamp":"2024-05-15T13:10:17.051453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465988,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":170,"num_write_kb":1429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133,"ondisk_log_size":133,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":577092,"kb_used_data":43028,"kb_used_omap":12,"kb_used_meta":533939,"kb_avail":754397628,"statfs":{"total":773094113280,"available":772503171072,"internally_reserved":0,"allocated":44060672,"data_stored":22717930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":546754096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":192,"apply_latency_ms":192,"commit_latency_ns":192000000,"apply_latency_ns":192000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.411676"},"pg_stats":[{"pgid":"64.1f","version":"189'3","reported_seq":2377,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.159746+0000","last_change":"2024-05-15T12:35:50.056262+0000","last_active":"2024-05-15T13:10:15.159746+0000","last_peered":"2024-05-15T13:10:15.159746+0000","last_clean":"2024-05-15T13:10:15.159746+0000","last_became_active":"2024-05-15T12:35:49.922024+0000","last_became_peered":"2024-05-15T12:35:49.922024+0000","last_unstale":"2024-05-15T13:10:15.159746+0000","last_undegraded":"2024-05-15T13:10:15.159746+0000","last_fullsized":"2024-05-15T13:10:15.159746+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:51:48.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1e","version":"0'0","reported_seq":2366,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.766043+0000","last_change":"2024-05-15T12:35:32.331096+0000","last_active":"2024-05-15T13:10:11.766043+0000","last_peered":"2024-05-15T13:10:11.766043+0000","last_clean":"2024-05-15T13:10:11.766043+0000","last_became_active":"2024-05-15T12:35:32.314677+0000","last_became_peered":"2024-05-15T12:35:32.314677+0000","last_unstale":"2024-05-15T13:10:11.766043+0000","last_undegraded":"2024-05-15T13:10:11.766043+0000","last_fullsized":"2024-05-15T13:10:11.766043+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:14:42.373968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1d","version":"190'2","reported_seq":2356,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.302418+0000","last_change":"2024-05-15T12:36:16.192838+0000","last_active":"2024-05-15T13:10:13.302418+0000","last_peered":"2024-05-15T13:10:13.302418+0000","last_clean":"2024-05-15T13:10:13.302418+0000","last_became_active":"2024-05-15T12:36:16.182561+0000","last_became_peered":"2024-05-15T12:36:16.182561+0000","last_unstale":"2024-05-15T13:10:13.302418+0000","last_undegraded":"2024-05-15T13:10:13.302418+0000","last_fullsized":"2024-05-15T13:10:13.302418+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:22:35.686460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1c","version":"0'0","reported_seq":2373,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.123202+0000","last_change":"2024-05-15T12:35:47.611774+0000","last_active":"2024-05-15T13:10:15.123202+0000","last_peered":"2024-05-15T13:10:15.123202+0000","last_clean":"2024-05-15T13:10:15.123202+0000","last_became_active":"2024-05-15T12:35:47.561938+0000","last_became_peered":"2024-05-15T12:35:47.561938+0000","last_unstale":"2024-05-15T13:10:15.123202+0000","last_undegraded":"2024-05-15T13:10:15.123202+0000","last_fullsized":"2024-05-15T13:10:15.123202+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:53:18.771807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1b","version":"163'1","reported_seq":2375,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.806601+0000","last_change":"2024-05-15T12:35:31.343846+0000","last_active":"2024-05-15T13:10:11.806601+0000","last_peered":"2024-05-15T13:10:11.806601+0000","last_clean":"2024-05-15T13:10:11.806601+0000","last_became_active":"2024-05-15T12:35:31.337467+0000","last_became_peered":"2024-05-15T12:35:31.337467+0000","last_unstale":"2024-05-15T13:10:11.806601+0000","last_undegraded":"2024-05-15T13:10:11.806601+0000","last_fullsized":"2024-05-15T13:10:11.806601+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:43:08.695881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.1a","version":"190'4","reported_seq":2359,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.284412+0000","last_change":"2024-05-15T12:35:44.341452+0000","last_active":"2024-05-15T13:10:13.284412+0000","last_peered":"2024-05-15T13:10:13.284412+0000","last_clean":"2024-05-15T13:10:13.284412+0000","last_became_active":"2024-05-15T12:35:44.321742+0000","last_became_peered":"2024-05-15T12:35:44.321742+0000","last_unstale":"2024-05-15T13:10:13.284412+0000","last_undegraded":"2024-05-15T13:10:13.284412+0000","last_fullsized":"2024-05-15T13:10:13.284412+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:47:56.382044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.19","version":"190'1","reported_seq":2375,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.104134+0000","last_change":"2024-05-15T12:35:49.203007+0000","last_active":"2024-05-15T13:10:15.104134+0000","last_peered":"2024-05-15T13:10:15.104134+0000","last_clean":"2024-05-15T13:10:15.104134+0000","last_became_active":"2024-05-15T12:35:49.195589+0000","last_became_peered":"2024-05-15T12:35:49.195589+0000","last_unstale":"2024-05-15T13:10:15.104134+0000","last_undegraded":"2024-05-15T13:10:15.104134+0000","last_fullsized":"2024-05-15T13:10:15.104134+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:31:42.949909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.18","version":"165'3","reported_seq":2358,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.238393+0000","last_change":"2024-05-15T12:35:44.420062+0000","last_active":"2024-05-15T13:10:13.238393+0000","last_peered":"2024-05-15T13:10:13.238393+0000","last_clean":"2024-05-15T13:10:13.238393+0000","last_became_active":"2024-05-15T12:35:44.403250+0000","last_became_peered":"2024-05-15T12:35:44.403250+0000","last_unstale":"2024-05-15T13:10:13.238393+0000","last_undegraded":"2024-05-15T13:10:13.238393+0000","last_fullsized":"2024-05-15T13:10:13.238393+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:32:30.979766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.17","version":"190'2","reported_seq":2369,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.285796+0000","last_change":"2024-05-15T12:35:50.527941+0000","last_active":"2024-05-15T13:10:16.285796+0000","last_peered":"2024-05-15T13:10:16.285796+0000","last_clean":"2024-05-15T13:10:16.285796+0000","last_became_active":"2024-05-15T12:35:50.518230+0000","last_became_peered":"2024-05-15T12:35:50.518230+0000","last_unstale":"2024-05-15T13:10:16.285796+0000","last_undegraded":"2024-05-15T13:10:16.285796+0000","last_fullsized":"2024-05-15T13:10:16.285796+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:55:42.493548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.16","version":"180'4","reported_seq":2351,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.767988+0000","last_change":"2024-05-15T12:35:47.452786+0000","last_active":"2024-05-15T13:10:11.767988+0000","last_peered":"2024-05-15T13:10:11.767988+0000","last_clean":"2024-05-15T13:10:11.767988+0000","last_became_active":"2024-05-15T12:35:47.446210+0000","last_became_peered":"2024-05-15T12:35:47.446210+0000","last_unstale":"2024-05-15T13:10:11.767988+0000","last_undegraded":"2024-05-15T13:10:11.767988+0000","last_fullsized":"2024-05-15T13:10:11.767988+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:10:41.816557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.15","version":"190'2","reported_seq":2376,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.146313+0000","last_change":"2024-05-15T12:35:49.852126+0000","last_active":"2024-05-15T13:10:15.146313+0000","last_peered":"2024-05-15T13:10:15.146313+0000","last_clean":"2024-05-15T13:10:15.146313+0000","last_became_active":"2024-05-15T12:35:49.823395+0000","last_became_peered":"2024-05-15T12:35:49.823395+0000","last_unstale":"2024-05-15T13:10:15.146313+0000","last_undegraded":"2024-05-15T13:10:15.146313+0000","last_fullsized":"2024-05-15T13:10:15.146313+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:35:00.942178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.14","version":"0'0","reported_seq":2361,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.795951+0000","last_change":"2024-05-15T12:35:47.935648+0000","last_active":"2024-05-15T13:10:11.795951+0000","last_peered":"2024-05-15T13:10:11.795951+0000","last_clean":"2024-05-15T13:10:11.795951+0000","last_became_active":"2024-05-15T12:35:47.926796+0000","last_became_peered":"2024-05-15T12:35:47.926796+0000","last_unstale":"2024-05-15T13:10:11.795951+0000","last_undegraded":"2024-05-15T13:10:11.795951+0000","last_fullsized":"2024-05-15T13:10:11.795951+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:12:08.277421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.13","version":"171'1","reported_seq":2376,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.810073+0000","last_change":"2024-05-15T12:35:28.188222+0000","last_active":"2024-05-15T13:10:11.810073+0000","last_peered":"2024-05-15T13:10:11.810073+0000","last_clean":"2024-05-15T13:10:11.810073+0000","last_became_active":"2024-05-15T12:35:28.182751+0000","last_became_peered":"2024-05-15T12:35:28.182751+0000","last_unstale":"2024-05-15T13:10:11.810073+0000","last_undegraded":"2024-05-15T13:10:11.810073+0000","last_fullsized":"2024-05-15T13:10:11.810073+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:38:40.701230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.12","version":"79'1","reported_seq":2380,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.259423+0000","last_change":"2024-05-15T12:35:27.725919+0000","last_active":"2024-05-15T13:10:16.259423+0000","last_peered":"2024-05-15T13:10:16.259423+0000","last_clean":"2024-05-15T13:10:16.259423+0000","last_became_active":"2024-05-15T12:35:27.719548+0000","last_became_peered":"2024-05-15T12:35:27.719548+0000","last_unstale":"2024-05-15T13:10:16.259423+0000","last_undegraded":"2024-05-15T13:10:16.259423+0000","last_fullsized":"2024-05-15T13:10:16.259423+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:26:15.224906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.11","version":"172'1","reported_seq":2370,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.246403+0000","last_change":"2024-05-15T12:35:50.434980+0000","last_active":"2024-05-15T13:10:16.246403+0000","last_peered":"2024-05-15T13:10:16.246403+0000","last_clean":"2024-05-15T13:10:16.246403+0000","last_became_active":"2024-05-15T12:35:50.426351+0000","last_became_peered":"2024-05-15T12:35:50.426351+0000","last_unstale":"2024-05-15T13:10:16.246403+0000","last_undegraded":"2024-05-15T13:10:16.246403+0000","last_fullsized":"2024-05-15T13:10:16.246403+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:11:28.596763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.10","version":"190'3","reported_seq":2361,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.735401+0000","last_change":"2024-05-15T12:35:34.592833+0000","last_active":"2024-05-15T13:10:11.735401+0000","last_peered":"2024-05-15T13:10:11.735401+0000","last_clean":"2024-05-15T13:10:11.735401+0000","last_became_active":"2024-05-15T12:35:34.587282+0000","last_became_peered":"2024-05-15T12:35:34.587282+0000","last_unstale":"2024-05-15T13:10:11.735401+0000","last_undegraded":"2024-05-15T13:10:11.735401+0000","last_fullsized":"2024-05-15T13:10:11.735401+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:35.500859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.f","version":"155'2","reported_seq":2357,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.282099+0000","last_change":"2024-05-15T12:35:47.741135+0000","last_active":"2024-05-15T13:10:13.282099+0000","last_peered":"2024-05-15T13:10:13.282099+0000","last_clean":"2024-05-15T13:10:13.282099+0000","last_became_active":"2024-05-15T12:35:47.063782+0000","last_became_peered":"2024-05-15T12:35:47.063782+0000","last_unstale":"2024-05-15T13:10:13.282099+0000","last_undegraded":"2024-05-15T13:10:13.282099+0000","last_fullsized":"2024-05-15T13:10:13.282099+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:05:49.823797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.e","version":"174'1","reported_seq":2380,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.272871+0000","last_change":"2024-05-15T12:35:28.039423+0000","last_active":"2024-05-15T13:10:16.272871+0000","last_peered":"2024-05-15T13:10:16.272871+0000","last_clean":"2024-05-15T13:10:16.272871+0000","last_became_active":"2024-05-15T12:35:28.033694+0000","last_became_peered":"2024-05-15T12:35:28.033694+0000","last_unstale":"2024-05-15T13:10:16.272871+0000","last_undegraded":"2024-05-15T13:10:16.272871+0000","last_fullsized":"2024-05-15T13:10:16.272871+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:13:51.876959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.d","version":"175'3","reported_seq":2371,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.262866+0000","last_change":"2024-05-15T12:35:50.592101+0000","last_active":"2024-05-15T13:10:16.262866+0000","last_peered":"2024-05-15T13:10:16.262866+0000","last_clean":"2024-05-15T13:10:16.262866+0000","last_became_active":"2024-05-15T12:35:50.586234+0000","last_became_peered":"2024-05-15T12:35:50.586234+0000","last_unstale":"2024-05-15T13:10:16.262866+0000","last_undegraded":"2024-05-15T13:10:16.262866+0000","last_fullsized":"2024-05-15T13:10:16.262866+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:23:52.114251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.c","version":"190'1","reported_seq":2370,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.080944+0000","last_change":"2024-05-15T12:35:39.569380+0000","last_active":"2024-05-15T13:10:14.080944+0000","last_peered":"2024-05-15T13:10:14.080944+0000","last_clean":"2024-05-15T13:10:14.080944+0000","last_became_active":"2024-05-15T12:35:39.550911+0000","last_became_peered":"2024-05-15T12:35:39.550911+0000","last_unstale":"2024-05-15T13:10:14.080944+0000","last_undegraded":"2024-05-15T13:10:14.080944+0000","last_fullsized":"2024-05-15T13:10:14.080944+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:11:58.527351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.b","version":"186'2","reported_seq":2365,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.748051+0000","last_change":"2024-05-15T12:35:22.786550+0000","last_active":"2024-05-15T13:10:11.748051+0000","last_peered":"2024-05-15T13:10:11.748051+0000","last_clean":"2024-05-15T13:10:11.748051+0000","last_became_active":"2024-05-15T12:35:22.781126+0000","last_became_peered":"2024-05-15T12:35:22.781126+0000","last_unstale":"2024-05-15T13:10:11.748051+0000","last_undegraded":"2024-05-15T13:10:11.748051+0000","last_fullsized":"2024-05-15T13:10:11.748051+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:11:51.197065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.a","version":"170'1","reported_seq":2364,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.778652+0000","last_change":"2024-05-15T12:35:47.948588+0000","last_active":"2024-05-15T13:10:11.778652+0000","last_peered":"2024-05-15T13:10:11.778652+0000","last_clean":"2024-05-15T13:10:11.778652+0000","last_became_active":"2024-05-15T12:35:47.943168+0000","last_became_peered":"2024-05-15T12:35:47.943168+0000","last_unstale":"2024-05-15T13:10:11.778652+0000","last_undegraded":"2024-05-15T13:10:11.778652+0000","last_fullsized":"2024-05-15T13:10:11.778652+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:00:16.296438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.9","version":"190'1","reported_seq":2377,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.107437+0000","last_change":"2024-05-15T12:35:49.112002+0000","last_active":"2024-05-15T13:10:15.107437+0000","last_peered":"2024-05-15T13:10:15.107437+0000","last_clean":"2024-05-15T13:10:15.107437+0000","last_became_active":"2024-05-15T12:35:49.106669+0000","last_became_peered":"2024-05-15T12:35:49.106669+0000","last_unstale":"2024-05-15T13:10:15.107437+0000","last_undegraded":"2024-05-15T13:10:15.107437+0000","last_fullsized":"2024-05-15T13:10:15.107437+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:32:01.416983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.8","version":"0'0","reported_seq":2381,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.236425+0000","last_change":"2024-05-15T12:35:27.822672+0000","last_active":"2024-05-15T13:10:16.236425+0000","last_peered":"2024-05-15T13:10:16.236425+0000","last_clean":"2024-05-15T13:10:16.236425+0000","last_became_active":"2024-05-15T12:35:27.816334+0000","last_became_peered":"2024-05-15T12:35:27.816334+0000","last_unstale":"2024-05-15T13:10:16.236425+0000","last_undegraded":"2024-05-15T13:10:16.236425+0000","last_fullsized":"2024-05-15T13:10:16.236425+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:10:34.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.7","version":"172'1","reported_seq":2375,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.161975+0000","last_change":"2024-05-15T12:35:47.775023+0000","last_active":"2024-05-15T13:10:15.161975+0000","last_peered":"2024-05-15T13:10:15.161975+0000","last_clean":"2024-05-15T13:10:15.161975+0000","last_became_active":"2024-05-15T12:35:47.766045+0000","last_became_peered":"2024-05-15T12:35:47.766045+0000","last_unstale":"2024-05-15T13:10:15.161975+0000","last_undegraded":"2024-05-15T13:10:15.161975+0000","last_fullsized":"2024-05-15T13:10:15.161975+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:35:16.103745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.6","version":"0'0","reported_seq":2377,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.056994+0000","last_change":"2024-05-15T12:35:59.068641+0000","last_active":"2024-05-15T13:10:14.056994+0000","last_peered":"2024-05-15T13:10:14.056994+0000","last_clean":"2024-05-15T13:10:14.056994+0000","last_became_active":"2024-05-15T12:35:59.010795+0000","last_became_peered":"2024-05-15T12:35:59.010795+0000","last_unstale":"2024-05-15T13:10:14.056994+0000","last_undegraded":"2024-05-15T13:10:14.056994+0000","last_fullsized":"2024-05-15T13:10:14.056994+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:26:58.973165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.5","version":"161'1","reported_seq":2357,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.304715+0000","last_change":"2024-05-15T12:36:31.757984+0000","last_active":"2024-05-15T13:10:13.304715+0000","last_peered":"2024-05-15T13:10:13.304715+0000","last_clean":"2024-05-15T13:10:13.304715+0000","last_became_active":"2024-05-15T12:36:31.177992+0000","last_became_peered":"2024-05-15T12:36:31.177992+0000","last_unstale":"2024-05-15T13:10:13.304715+0000","last_undegraded":"2024-05-15T13:10:13.304715+0000","last_fullsized":"2024-05-15T13:10:13.304715+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:51:21.937298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.4","version":"0'0","reported_seq":2375,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.047899+0000","last_change":"2024-05-15T12:35:59.243799+0000","last_active":"2024-05-15T13:10:14.047899+0000","last_peered":"2024-05-15T13:10:14.047899+0000","last_clean":"2024-05-15T13:10:14.047899+0000","last_became_active":"2024-05-15T12:35:59.238121+0000","last_became_peered":"2024-05-15T12:35:59.238121+0000","last_unstale":"2024-05-15T13:10:14.047899+0000","last_undegraded":"2024-05-15T13:10:14.047899+0000","last_fullsized":"2024-05-15T13:10:14.047899+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:44:24.673671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.3","version":"190'4","reported_seq":2379,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.111755+0000","last_change":"2024-05-15T12:35:47.667458+0000","last_active":"2024-05-15T13:10:15.111755+0000","last_peered":"2024-05-15T13:10:15.111755+0000","last_clean":"2024-05-15T13:10:15.111755+0000","last_became_active":"2024-05-15T12:35:47.639452+0000","last_became_peered":"2024-05-15T12:35:47.639452+0000","last_unstale":"2024-05-15T13:10:15.111755+0000","last_undegraded":"2024-05-15T13:10:15.111755+0000","last_fullsized":"2024-05-15T13:10:15.111755+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:00:58.006820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":704,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.2","version":"143'1","reported_seq":2389,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.037872+0000","last_change":"2024-05-15T12:35:32.749852+0000","last_active":"2024-05-15T13:10:14.037872+0000","last_peered":"2024-05-15T13:10:14.037872+0000","last_clean":"2024-05-15T13:10:14.037872+0000","last_became_active":"2024-05-15T12:35:32.699857+0000","last_became_peered":"2024-05-15T12:35:32.699857+0000","last_unstale":"2024-05-15T13:10:14.037872+0000","last_undegraded":"2024-05-15T13:10:14.037872+0000","last_fullsized":"2024-05-15T13:10:14.037872+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:24:48.769777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1","version":"190'3","reported_seq":2360,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.298789+0000","last_change":"2024-05-15T12:36:13.620976+0000","last_active":"2024-05-15T13:10:13.298789+0000","last_peered":"2024-05-15T13:10:13.298789+0000","last_clean":"2024-05-15T13:10:13.298789+0000","last_became_active":"2024-05-15T12:36:13.614839+0000","last_became_peered":"2024-05-15T12:36:13.614839+0000","last_unstale":"2024-05-15T13:10:13.298789+0000","last_undegraded":"2024-05-15T13:10:13.298789+0000","last_fullsized":"2024-05-15T13:10:13.298789+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:01:15.571189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.0","version":"166'1","reported_seq":2352,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.741098+0000","last_change":"2024-05-15T12:35:47.277656+0000","last_active":"2024-05-15T13:10:11.741098+0000","last_peered":"2024-05-15T13:10:11.741098+0000","last_clean":"2024-05-15T13:10:11.741098+0000","last_became_active":"2024-05-15T12:35:47.271050+0000","last_became_peered":"2024-05-15T12:35:47.271050+0000","last_unstale":"2024-05-15T13:10:11.741098+0000","last_undegraded":"2024-05-15T13:10:11.741098+0000","last_fullsized":"2024-05-15T13:10:11.741098+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:12:17.443633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":2399,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.279993+0000","last_change":"2024-05-15T12:34:57.169160+0000","last_active":"2024-05-15T13:10:13.279993+0000","last_peered":"2024-05-15T13:10:13.279993+0000","last_clean":"2024-05-15T13:10:13.279993+0000","last_became_active":"2024-05-15T12:34:57.154951+0000","last_became_peered":"2024-05-15T12:34:57.154951+0000","last_unstale":"2024-05-15T13:10:13.279993+0000","last_undegraded":"2024-05-15T13:10:13.279993+0000","last_fullsized":"2024-05-15T13:10:13.279993+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:37:02.028559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":2401,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.336386+0000","last_change":"2024-05-15T12:34:54.507835+0000","last_active":"2024-05-15T13:10:13.336386+0000","last_peered":"2024-05-15T13:10:13.336386+0000","last_clean":"2024-05-15T13:10:13.336386+0000","last_became_active":"2024-05-15T12:34:54.498165+0000","last_became_peered":"2024-05-15T12:34:54.498165+0000","last_unstale":"2024-05-15T13:10:13.336386+0000","last_undegraded":"2024-05-15T13:10:13.336386+0000","last_fullsized":"2024-05-15T13:10:13.336386+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:35:01.164627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":2411,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.265039+0000","last_change":"2024-05-15T12:34:52.684394+0000","last_active":"2024-05-15T13:10:16.265039+0000","last_peered":"2024-05-15T13:10:16.265039+0000","last_clean":"2024-05-15T13:10:16.265039+0000","last_became_active":"2024-05-15T12:34:52.677394+0000","last_became_peered":"2024-05-15T12:34:52.677394+0000","last_unstale":"2024-05-15T13:10:16.265039+0000","last_undegraded":"2024-05-15T13:10:16.265039+0000","last_fullsized":"2024-05-15T13:10:16.265039+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:10:31.152782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":2417,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.124924+0000","last_change":"2024-05-15T12:34:50.121401+0000","last_active":"2024-05-15T13:10:15.124924+0000","last_peered":"2024-05-15T13:10:15.124924+0000","last_clean":"2024-05-15T13:10:15.124924+0000","last_became_active":"2024-05-15T12:34:50.114215+0000","last_became_peered":"2024-05-15T12:34:50.114215+0000","last_unstale":"2024-05-15T13:10:15.124924+0000","last_undegraded":"2024-05-15T13:10:15.124924+0000","last_fullsized":"2024-05-15T13:10:15.124924+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:04:16.442442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":2413,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.939553+0000","last_change":"2024-05-15T12:34:54.772141+0000","last_active":"2024-05-15T13:10:13.939553+0000","last_peered":"2024-05-15T13:10:13.939553+0000","last_clean":"2024-05-15T13:10:13.939553+0000","last_became_active":"2024-05-15T12:34:54.762972+0000","last_became_peered":"2024-05-15T12:34:54.762972+0000","last_unstale":"2024-05-15T13:10:13.939553+0000","last_undegraded":"2024-05-15T13:10:13.939553+0000","last_fullsized":"2024-05-15T13:10:13.939553+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:01:23.482313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.780442+0000","last_change":"2024-05-15T12:34:34.684239+0000","last_active":"2024-05-15T13:10:11.780442+0000","last_peered":"2024-05-15T13:10:11.780442+0000","last_clean":"2024-05-15T13:10:11.780442+0000","last_became_active":"2024-05-15T12:34:34.670411+0000","last_became_peered":"2024-05-15T12:34:34.670411+0000","last_unstale":"2024-05-15T13:10:11.780442+0000","last_undegraded":"2024-05-15T13:10:11.780442+0000","last_fullsized":"2024-05-15T13:10:11.780442+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:15:28.642471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":2418,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.108453+0000","last_change":"2024-05-15T12:34:50.164567+0000","last_active":"2024-05-15T13:10:15.108453+0000","last_peered":"2024-05-15T13:10:15.108453+0000","last_clean":"2024-05-15T13:10:15.108453+0000","last_became_active":"2024-05-15T12:34:50.159068+0000","last_became_peered":"2024-05-15T12:34:50.159068+0000","last_unstale":"2024-05-15T13:10:15.108453+0000","last_undegraded":"2024-05-15T13:10:15.108453+0000","last_fullsized":"2024-05-15T13:10:15.108453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:05:11.053203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":2424,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.116855+0000","last_change":"2024-05-15T12:34:39.918992+0000","last_active":"2024-05-15T13:10:15.116855+0000","last_peered":"2024-05-15T13:10:15.116855+0000","last_clean":"2024-05-15T13:10:15.116855+0000","last_became_active":"2024-05-15T12:34:39.907050+0000","last_became_peered":"2024-05-15T12:34:39.907050+0000","last_unstale":"2024-05-15T13:10:15.116855+0000","last_undegraded":"2024-05-15T13:10:15.116855+0000","last_fullsized":"2024-05-15T13:10:15.116855+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:10:12.279986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":2392,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.796844+0000","last_change":"2024-05-15T12:34:44.459281+0000","last_active":"2024-05-15T13:10:11.796844+0000","last_peered":"2024-05-15T13:10:11.796844+0000","last_clean":"2024-05-15T13:10:11.796844+0000","last_became_active":"2024-05-15T12:34:44.443445+0000","last_became_peered":"2024-05-15T12:34:44.443445+0000","last_unstale":"2024-05-15T13:10:11.796844+0000","last_undegraded":"2024-05-15T13:10:11.796844+0000","last_fullsized":"2024-05-15T13:10:11.796844+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:03:29.028978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":2394,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.771857+0000","last_change":"2024-05-15T12:34:38.200148+0000","last_active":"2024-05-15T13:10:11.771857+0000","last_peered":"2024-05-15T13:10:11.771857+0000","last_clean":"2024-05-15T13:10:11.771857+0000","last_became_active":"2024-05-15T12:34:38.194477+0000","last_became_peered":"2024-05-15T12:34:38.194477+0000","last_unstale":"2024-05-15T13:10:11.771857+0000","last_undegraded":"2024-05-15T13:10:11.771857+0000","last_fullsized":"2024-05-15T13:10:11.771857+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:28:19.294534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":2400,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.306918+0000","last_change":"2024-05-15T12:34:42.333270+0000","last_active":"2024-05-15T13:10:13.306918+0000","last_peered":"2024-05-15T13:10:13.306918+0000","last_clean":"2024-05-15T13:10:13.306918+0000","last_became_active":"2024-05-15T12:34:42.317054+0000","last_became_peered":"2024-05-15T12:34:42.317054+0000","last_unstale":"2024-05-15T13:10:13.306918+0000","last_undegraded":"2024-05-15T13:10:13.306918+0000","last_fullsized":"2024-05-15T13:10:13.306918+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:15:51.004019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":2411,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.078980+0000","last_change":"2024-05-15T12:34:48.752104+0000","last_active":"2024-05-15T13:10:14.078980+0000","last_peered":"2024-05-15T13:10:14.078980+0000","last_clean":"2024-05-15T13:10:14.078980+0000","last_became_active":"2024-05-15T12:34:48.742068+0000","last_became_peered":"2024-05-15T12:34:48.742068+0000","last_unstale":"2024-05-15T13:10:14.078980+0000","last_undegraded":"2024-05-15T13:10:14.078980+0000","last_fullsized":"2024-05-15T13:10:14.078980+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:18:38.517375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"26'76","reported_seq":2461,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.243120+0000","last_change":"2024-05-15T12:32:43.562093+0000","last_active":"2024-05-15T13:10:13.243120+0000","last_peered":"2024-05-15T13:10:13.243120+0000","last_clean":"2024-05-15T13:10:13.243120+0000","last_became_active":"2024-05-15T12:32:42.889644+0000","last_became_peered":"2024-05-15T12:32:42.889644+0000","last_unstale":"2024-05-15T13:10:13.243120+0000","last_undegraded":"2024-05-15T13:10:13.243120+0000","last_fullsized":"2024-05-15T13:10:13.243120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:30.812143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:30.812143+0000","last_clean_scrub_stamp":"2024-05-15T12:32:30.812143+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:45:22.988060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2426,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.281969+0000","last_change":"2024-05-15T12:32:47.722217+0000","last_active":"2024-05-15T13:10:16.281969+0000","last_peered":"2024-05-15T13:10:16.281969+0000","last_clean":"2024-05-15T13:10:16.281969+0000","last_became_active":"2024-05-15T12:32:47.716674+0000","last_became_peered":"2024-05-15T12:32:47.716674+0000","last_unstale":"2024-05-15T13:10:16.281969+0000","last_undegraded":"2024-05-15T13:10:16.281969+0000","last_fullsized":"2024-05-15T13:10:16.281969+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:25:03.820946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":2423,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.142241+0000","last_change":"2024-05-15T12:34:42.947741+0000","last_active":"2024-05-15T13:10:15.142241+0000","last_peered":"2024-05-15T13:10:15.142241+0000","last_clean":"2024-05-15T13:10:15.142241+0000","last_became_active":"2024-05-15T12:34:42.942214+0000","last_became_peered":"2024-05-15T12:34:42.942214+0000","last_unstale":"2024-05-15T13:10:15.142241+0000","last_undegraded":"2024-05-15T13:10:15.142241+0000","last_fullsized":"2024-05-15T13:10:15.142241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:08:22.005787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2427,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.238570+0000","last_change":"2024-05-15T12:32:47.459103+0000","last_active":"2024-05-15T13:10:16.238570+0000","last_peered":"2024-05-15T13:10:16.238570+0000","last_clean":"2024-05-15T13:10:16.238570+0000","last_became_active":"2024-05-15T12:32:47.430194+0000","last_became_peered":"2024-05-15T12:32:47.430194+0000","last_unstale":"2024-05-15T13:10:16.238570+0000","last_undegraded":"2024-05-15T13:10:16.238570+0000","last_fullsized":"2024-05-15T13:10:16.238570+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:14:58.294854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":2414,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.267919+0000","last_change":"2024-05-15T12:34:52.363872+0000","last_active":"2024-05-15T13:10:16.267919+0000","last_peered":"2024-05-15T13:10:16.267919+0000","last_clean":"2024-05-15T13:10:16.267919+0000","last_became_active":"2024-05-15T12:34:52.357328+0000","last_became_peered":"2024-05-15T12:34:52.357328+0000","last_unstale":"2024-05-15T13:10:16.267919+0000","last_undegraded":"2024-05-15T13:10:16.267919+0000","last_fullsized":"2024-05-15T13:10:16.267919+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:05:48.456769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2408,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.743300+0000","last_change":"2024-05-15T12:32:47.463044+0000","last_active":"2024-05-15T13:10:11.743300+0000","last_peered":"2024-05-15T13:10:11.743300+0000","last_clean":"2024-05-15T13:10:11.743300+0000","last_became_active":"2024-05-15T12:32:47.432679+0000","last_became_peered":"2024-05-15T12:32:47.432679+0000","last_unstale":"2024-05-15T13:10:11.743300+0000","last_undegraded":"2024-05-15T13:10:11.743300+0000","last_fullsized":"2024-05-15T13:10:11.743300+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:13:56.912246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":2414,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.800112+0000","last_change":"2024-05-15T12:34:36.929836+0000","last_active":"2024-05-15T13:10:11.800112+0000","last_peered":"2024-05-15T13:10:11.800112+0000","last_clean":"2024-05-15T13:10:11.800112+0000","last_became_active":"2024-05-15T12:34:36.924450+0000","last_became_peered":"2024-05-15T12:34:36.924450+0000","last_unstale":"2024-05-15T13:10:11.800112+0000","last_undegraded":"2024-05-15T13:10:11.800112+0000","last_fullsized":"2024-05-15T13:10:11.800112+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:13:01.098128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2429,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.256147+0000","last_change":"2024-05-15T12:36:58.748207+0000","last_active":"2024-05-15T13:10:16.256147+0000","last_peered":"2024-05-15T13:10:16.256147+0000","last_clean":"2024-05-15T13:10:16.256147+0000","last_became_active":"2024-05-15T12:32:47.670226+0000","last_became_peered":"2024-05-15T12:32:47.670226+0000","last_unstale":"2024-05-15T13:10:16.256147+0000","last_undegraded":"2024-05-15T13:10:16.256147+0000","last_fullsized":"2024-05-15T13:10:16.256147+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:41:09.623434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":2413,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.287949+0000","last_change":"2024-05-15T12:34:51.754656+0000","last_active":"2024-05-15T13:10:16.287949+0000","last_peered":"2024-05-15T13:10:16.287949+0000","last_clean":"2024-05-15T13:10:16.287949+0000","last_became_active":"2024-05-15T12:34:51.746853+0000","last_became_peered":"2024-05-15T12:34:51.746853+0000","last_unstale":"2024-05-15T13:10:16.287949+0000","last_undegraded":"2024-05-15T13:10:16.287949+0000","last_fullsized":"2024-05-15T13:10:16.287949+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:59:19.605051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2436,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.126699+0000","last_change":"2024-05-15T12:32:47.687489+0000","last_active":"2024-05-15T13:10:15.126699+0000","last_peered":"2024-05-15T13:10:15.126699+0000","last_clean":"2024-05-15T13:10:15.126699+0000","last_became_active":"2024-05-15T12:32:47.680498+0000","last_became_peered":"2024-05-15T12:32:47.680498+0000","last_unstale":"2024-05-15T13:10:15.126699+0000","last_undegraded":"2024-05-15T13:10:15.126699+0000","last_fullsized":"2024-05-15T13:10:15.126699+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:09:36.650810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"39'7","reported_seq":2427,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.048781+0000","last_change":"2024-05-15T12:34:50.381955+0000","last_active":"2024-05-15T13:10:14.048781+0000","last_peered":"2024-05-15T13:10:14.048781+0000","last_clean":"2024-05-15T13:10:14.048781+0000","last_became_active":"2024-05-15T12:34:50.371773+0000","last_became_peered":"2024-05-15T12:34:50.371773+0000","last_unstale":"2024-05-15T13:10:14.048781+0000","last_undegraded":"2024-05-15T13:10:14.048781+0000","last_fullsized":"2024-05-15T13:10:14.048781+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:40:48.381654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2425,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.268960+0000","last_change":"2024-05-15T12:32:47.531833+0000","last_active":"2024-05-15T13:10:16.268960+0000","last_peered":"2024-05-15T13:10:16.268960+0000","last_clean":"2024-05-15T13:10:16.268960+0000","last_became_active":"2024-05-15T12:32:47.526275+0000","last_became_peered":"2024-05-15T12:32:47.526275+0000","last_unstale":"2024-05-15T13:10:16.268960+0000","last_undegraded":"2024-05-15T13:10:16.268960+0000","last_fullsized":"2024-05-15T13:10:16.268960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:32:09.483290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":2404,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.241050+0000","last_change":"2024-05-15T12:34:38.895220+0000","last_active":"2024-05-15T13:10:13.241050+0000","last_peered":"2024-05-15T13:10:13.241050+0000","last_clean":"2024-05-15T13:10:13.241050+0000","last_became_active":"2024-05-15T12:34:38.888318+0000","last_became_peered":"2024-05-15T12:34:38.888318+0000","last_unstale":"2024-05-15T13:10:13.241050+0000","last_undegraded":"2024-05-15T13:10:13.241050+0000","last_fullsized":"2024-05-15T13:10:13.241050+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:38:34.358044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2424,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.818412+0000","last_change":"2024-05-15T12:32:48.309674+0000","last_active":"2024-05-15T13:10:11.818412+0000","last_peered":"2024-05-15T13:10:11.818412+0000","last_clean":"2024-05-15T13:10:11.818412+0000","last_became_active":"2024-05-15T12:32:48.283635+0000","last_became_peered":"2024-05-15T12:32:48.283635+0000","last_unstale":"2024-05-15T13:10:11.818412+0000","last_undegraded":"2024-05-15T13:10:11.818412+0000","last_fullsized":"2024-05-15T13:10:11.818412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:07:32.002393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":2424,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.113848+0000","last_change":"2024-05-15T12:34:48.672942+0000","last_active":"2024-05-15T13:10:15.113848+0000","last_peered":"2024-05-15T13:10:15.113848+0000","last_clean":"2024-05-15T13:10:15.113848+0000","last_became_active":"2024-05-15T12:34:48.664879+0000","last_became_peered":"2024-05-15T12:34:48.664879+0000","last_unstale":"2024-05-15T13:10:15.113848+0000","last_undegraded":"2024-05-15T13:10:15.113848+0000","last_fullsized":"2024-05-15T13:10:15.113848+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:48:54.955944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2424,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.289898+0000","last_change":"2024-05-15T12:32:48.258451+0000","last_active":"2024-05-15T13:10:16.289898+0000","last_peered":"2024-05-15T13:10:16.289898+0000","last_clean":"2024-05-15T13:10:16.289898+0000","last_became_active":"2024-05-15T12:32:48.252725+0000","last_became_peered":"2024-05-15T12:32:48.252725+0000","last_unstale":"2024-05-15T13:10:16.289898+0000","last_undegraded":"2024-05-15T13:10:16.289898+0000","last_fullsized":"2024-05-15T13:10:16.289898+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:01:52.775805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":2414,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.782737+0000","last_change":"2024-05-15T12:34:34.676904+0000","last_active":"2024-05-15T13:10:11.782737+0000","last_peered":"2024-05-15T13:10:11.782737+0000","last_clean":"2024-05-15T13:10:11.782737+0000","last_became_active":"2024-05-15T12:34:34.660678+0000","last_became_peered":"2024-05-15T12:34:34.660678+0000","last_unstale":"2024-05-15T13:10:11.782737+0000","last_undegraded":"2024-05-15T13:10:11.782737+0000","last_fullsized":"2024-05-15T13:10:11.782737+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:39:07.124084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":2394,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.739089+0000","last_change":"2024-05-15T12:34:44.299228+0000","last_active":"2024-05-15T13:10:11.739089+0000","last_peered":"2024-05-15T13:10:11.739089+0000","last_clean":"2024-05-15T13:10:11.739089+0000","last_became_active":"2024-05-15T12:34:44.277750+0000","last_became_peered":"2024-05-15T12:34:44.277750+0000","last_unstale":"2024-05-15T13:10:11.739089+0000","last_undegraded":"2024-05-15T13:10:11.739089+0000","last_fullsized":"2024-05-15T13:10:11.739089+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:00:02.340970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":2412,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.046745+0000","last_change":"2024-05-15T12:34:39.506099+0000","last_active":"2024-05-15T13:10:14.046745+0000","last_peered":"2024-05-15T13:10:14.046745+0000","last_clean":"2024-05-15T13:10:14.046745+0000","last_became_active":"2024-05-15T12:34:39.369183+0000","last_became_peered":"2024-05-15T12:34:39.369183+0000","last_unstale":"2024-05-15T13:10:14.046745+0000","last_undegraded":"2024-05-15T13:10:14.046745+0000","last_fullsized":"2024-05-15T13:10:14.046745+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:41:03.408587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":2402,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-05-15T13:10:13.286633+0000","last_change":"2024-05-15T12:34:38.831156+0000","last_active":"2024-05-15T13:10:13.286633+0000","last_peered":"2024-05-15T13:10:13.286633+0000","last_clean":"2024-05-15T13:10:13.286633+0000","last_became_active":"2024-05-15T12:34:38.812314+0000","last_became_peered":"2024-05-15T12:34:38.812314+0000","last_unstale":"2024-05-15T13:10:13.286633+0000","last_undegraded":"2024-05-15T13:10:13.286633+0000","last_fullsized":"2024-05-15T13:10:13.286633+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:45:31.161837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":2393,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.745771+0000","last_change":"2024-05-15T12:34:36.784260+0000","last_active":"2024-05-15T13:10:11.745771+0000","last_peered":"2024-05-15T13:10:11.745771+0000","last_clean":"2024-05-15T13:10:11.745771+0000","last_became_active":"2024-05-15T12:34:36.777340+0000","last_became_peered":"2024-05-15T12:34:36.777340+0000","last_unstale":"2024-05-15T13:10:11.745771+0000","last_undegraded":"2024-05-15T13:10:11.745771+0000","last_fullsized":"2024-05-15T13:10:11.745771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:27:38.620258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":2410,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.077032+0000","last_change":"2024-05-15T12:34:51.056513+0000","last_active":"2024-05-15T13:10:14.077032+0000","last_peered":"2024-05-15T13:10:14.077032+0000","last_clean":"2024-05-15T13:10:14.077032+0000","last_became_active":"2024-05-15T12:34:51.040048+0000","last_became_peered":"2024-05-15T12:34:51.040048+0000","last_unstale":"2024-05-15T13:10:14.077032+0000","last_undegraded":"2024-05-15T13:10:14.077032+0000","last_fullsized":"2024-05-15T13:10:14.077032+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:32:15.066105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":2422,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:15.148306+0000","last_change":"2024-05-15T12:34:51.656110+0000","last_active":"2024-05-15T13:10:15.148306+0000","last_peered":"2024-05-15T13:10:15.148306+0000","last_clean":"2024-05-15T13:10:15.148306+0000","last_became_active":"2024-05-15T12:34:51.649910+0000","last_became_peered":"2024-05-15T12:34:51.649910+0000","last_unstale":"2024-05-15T13:10:15.148306+0000","last_undegraded":"2024-05-15T13:10:15.148306+0000","last_fullsized":"2024-05-15T13:10:15.148306+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:01:47.425779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":2396,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.770003+0000","last_change":"2024-05-15T12:34:44.408851+0000","last_active":"2024-05-15T13:10:11.770003+0000","last_peered":"2024-05-15T13:10:11.770003+0000","last_clean":"2024-05-15T13:10:11.770003+0000","last_became_active":"2024-05-15T12:34:44.399991+0000","last_became_peered":"2024-05-15T12:34:44.399991+0000","last_unstale":"2024-05-15T13:10:11.770003+0000","last_undegraded":"2024-05-15T13:10:11.770003+0000","last_fullsized":"2024-05-15T13:10:11.770003+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:40:47.114950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":2424,"reported_epoch":1249,"state":"active+clean","last_fresh":"2024-05-15T13:10:14.064927+0000","last_change":"2024-05-15T12:34:38.557851+0000","last_active":"2024-05-15T13:10:14.064927+0000","last_peered":"2024-05-15T13:10:14.064927+0000","last_clean":"2024-05-15T13:10:14.064927+0000","last_became_active":"2024-05-15T12:34:38.549385+0000","last_became_peered":"2024-05-15T12:34:38.549385+0000","last_unstale":"2024-05-15T13:10:14.064927+0000","last_undegraded":"2024-05-15T13:10:14.064927+0000","last_fullsized":"2024-05-15T13:10:14.064927+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:31:24.444568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":2411,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.756882+0000","last_change":"2024-05-15T12:34:36.889179+0000","last_active":"2024-05-15T13:10:11.756882+0000","last_peered":"2024-05-15T13:10:11.756882+0000","last_clean":"2024-05-15T13:10:11.756882+0000","last_became_active":"2024-05-15T12:34:36.875596+0000","last_became_peered":"2024-05-15T12:34:36.875596+0000","last_unstale":"2024-05-15T13:10:11.756882+0000","last_undegraded":"2024-05-15T13:10:11.756882+0000","last_fullsized":"2024-05-15T13:10:11.756882+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:30:10.134778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":2413,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.244402+0000","last_change":"2024-05-15T12:34:51.659524+0000","last_active":"2024-05-15T13:10:16.244402+0000","last_peered":"2024-05-15T13:10:16.244402+0000","last_clean":"2024-05-15T13:10:16.244402+0000","last_became_active":"2024-05-15T12:34:51.653558+0000","last_became_peered":"2024-05-15T12:34:51.653558+0000","last_unstale":"2024-05-15T13:10:16.244402+0000","last_undegraded":"2024-05-15T13:10:16.244402+0000","last_fullsized":"2024-05-15T13:10:16.244402+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:43:01.117943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":2413,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-05-15T13:10:16.254206+0000","last_change":"2024-05-15T12:34:36.637973+0000","last_active":"2024-05-15T13:10:16.254206+0000","last_peered":"2024-05-15T13:10:16.254206+0000","last_clean":"2024-05-15T13:10:16.254206+0000","last_became_active":"2024-05-15T12:34:36.629372+0000","last_became_peered":"2024-05-15T12:34:36.629372+0000","last_unstale":"2024-05-15T13:10:16.254206+0000","last_undegraded":"2024-05-15T13:10:16.254206+0000","last_fullsized":"2024-05-15T13:10:16.254206+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:10:32.079108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":2408,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-05-15T13:10:11.812762+0000","last_change":"2024-05-15T12:34:49.358758+0000","last_active":"2024-05-15T13:10:11.812762+0000","last_peered":"2024-05-15T13:10:11.812762+0000","last_clean":"2024-05-15T13:10:11.812762+0000","last_became_active":"2024-05-15T12:34:49.324761+0000","last_became_peered":"2024-05-15T12:34:49.324761+0000","last_unstale":"2024-05-15T13:10:11.812762+0000","last_undegraded":"2024-05-15T13:10:11.812762+0000","last_fullsized":"2024-05-15T13:10:11.812762+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:13:55.863763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":6703,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":13406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313668,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68824,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303016,"statfs":{"total":96636764160,"available":96566288384,"internally_reserved":0,"allocated":5840896,"data_stored":3135087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[]},{"osd":7,"up_from":19,"seq":81604379073,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72528,"kb_used_data":5252,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299312,"statfs":{"total":96636764160,"available":96562495488,"internally_reserved":0,"allocated":5378048,"data_stored":2731941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":19,"seq":81604379080,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72980,"kb_used_data":5700,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298860,"statfs":{"total":96636764160,"available":96562032640,"internally_reserved":0,"allocated":5836800,"data_stored":3149637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444485,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76712,"kb_used_data":5272,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295128,"statfs":{"total":96636764160,"available":96558211072,"internally_reserved":0,"allocated":5398528,"data_stored":2735851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444483,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72528,"kb_used_data":5248,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299312,"statfs":{"total":96636764160,"available":96562495488,"internally_reserved":0,"allocated":5373952,"data_stored":2742021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542599,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72580,"kb_used_data":5304,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299260,"statfs":{"total":96636764160,"available":96562442240,"internally_reserved":0,"allocated":5431296,"data_stored":2744265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68412,"kb_used_data":5296,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303428,"statfs":{"total":96636764160,"available":96566710272,"internally_reserved":0,"allocated":5423104,"data_stored":2751231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575301,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72528,"kb_used_data":5252,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299312,"statfs":{"total":96636764160,"available":96562495488,"internally_reserved":0,"allocated":5378048,"data_stored":2727897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:10:17.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-15T13:10:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.669+0000 7fcfd2f64640 10 monclient: get_monmap_and_config 2024-05-15T13:10:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.669+0000 7fcfd2f64640 10 monclient: build_initial_monmap 2024-05-15T13:10:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.669+0000 7fcfd2f64640 10 monclient: monmap: 2024-05-15T13:10:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:17.669750+0000 2024-05-15T13:10:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:17.669750+0000 2024-05-15T13:10:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:17.660 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 1 Processor -- start 2024-05-15T13:10:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 1 -- start start 2024-05-15T13:10:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 10 monclient: init 2024-05-15T13:10:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-05-15T13:10:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcfcc0b53e0 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcfcc142870 0x7fcfcc142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 10 monclient(hunting): picked mon.noname-c con 0x7fcfcc142870 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcfcc1431f0 0x7fcfcc13f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 10 monclient(hunting): picked mon.noname-a con 0x7fcfcc1431f0 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfcc0ab230 con 0x7fcfcc0b53e0 2024-05-15T13:10:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfcc0b57c0 con 0x7fcfcc1431f0 2024-05-15T13:10:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfcc0aa230 con 0x7fcfcc142870 2024-05-15T13:10:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.670+0000 7fcfd2f64640 10 monclient(hunting): authenticate will time out at 3775.832297s 2024-05-15T13:10:17.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.677+0000 7fcfd0d00640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fcfcc0b53e0 0x7fcfcc0a99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:55236/0 (socket says 172.21.15.49:55236) 2024-05-15T13:10:17.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.677+0000 7fcfd0d00640 1 -- 172.21.15.49:0/687919547 learned_addr learned my addr 172.21.15.49:0/687919547 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:17.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.677+0000 7fcfd1501640 1 --2- 172.21.15.49:0/687919547 >> v2:172.21.15.49:3300/0 conn(0x7fcfcc1431f0 0x7fcfcc13f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.679+0000 7fcfd1501640 10 monclient(hunting): get_auth_request con 0x7fcfcc1431f0 auth_method 0 2024-05-15T13:10:17.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.679+0000 7fcfd1501640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:17.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.679+0000 7fcfd1501640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:17.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.679+0000 7fcfd1501640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:17.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.680+0000 7fcfcbfff640 1 --2- 172.21.15.49:0/687919547 >> v2:172.21.15.88:3300/0 conn(0x7fcfcc142870 0x7fcfcc142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.680+0000 7fcfcbfff640 10 monclient(hunting): get_auth_request con 0x7fcfcc142870 auth_method 0 2024-05-15T13:10:17.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.681+0000 7fcfcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:17.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.681+0000 7fcfcbfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:17.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.681+0000 7fcfcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:17.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.681+0000 7fcfcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:17.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.681+0000 7fcfcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:17.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.681+0000 7fcfcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:17.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.683+0000 7fcfcbfff640 10 monclient(hunting): handle_auth_done global_id 21997 payload 274 2024-05-15T13:10:17.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.683+0000 7fcfcbfff640 1 -- 172.21.15.49:0/687919547 >> v2:172.21.15.49:3300/0 conn(0x7fcfcc1431f0 msgr2=0x7fcfcc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:17.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.683+0000 7fcfcbfff640 1 --2- 172.21.15.49:0/687919547 >> v2:172.21.15.49:3300/0 conn(0x7fcfcc1431f0 0x7fcfcc13f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.683+0000 7fcfcbfff640 1 -- 172.21.15.49:0/687919547 >> v1:172.21.15.49:6789/0 conn(0x7fcfcc0b53e0 legacy=0x7fcfcc0a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.683+0000 7fcfcbfff640 10 monclient: _finish_hunting 0 2024-05-15T13:10:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.683+0000 7fcfcbfff640 1 monclient: found mon.noname-c 2024-05-15T13:10:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.683+0000 7fcfcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.683+0000 7fcfcbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T13:10:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.683+0000 7fcfcbfff640 1 -- 172.21.15.49:0/687919547 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc0ab510 con 0x7fcfcc142870 2024-05-15T13:10:17.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.686+0000 7fcfcbfff640 1 --2- 172.21.15.49:0/687919547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 0x7fcfcc142cb0 secure :-1 s=READY pgs=5445 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0b4660 tx=0x7fcfb802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e723094e3de759c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:17.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 1 -- 172.21.15.49:0/687919547 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80315a0 con 0x7fcfcc142870 2024-05-15T13:10:17.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:17.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:17.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 10 monclient: dump: 2024-05-15T13:10:17.686 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:17.687 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:17.687 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.687 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.687 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:17.687 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.688 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:17.688 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:17.688 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:17.688 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 10 monclient: _finish_auth 0 2024-05-15T13:10:17.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 10 monclient: _check_auth_tickets 2024-05-15T13:10:17.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:17.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 1 -- 172.21.15.49:0/687919547 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb8037070 con 0x7fcfcc142870 2024-05-15T13:10:17.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:17.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 1 -- 172.21.15.49:0/687919547 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8031960 con 0x7fcfcc142870 2024-05-15T13:10:17.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfd2f64640 5 monclient: authenticate success, global_id 21997 2024-05-15T13:10:17.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfd2f64640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:17.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfd2f64640 4 set_mon_vals no callback set 2024-05-15T13:10:17.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:17.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:17.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.690+0000 7fcfcb7fe640 10 monclient: dump: 2024-05-15T13:10:17.691 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:17.691 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:17.691 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.691 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.691 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:17.692 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.692 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:17.692 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:17.692 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:17.692 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.691+0000 7fcfd2f64640 10 monclient: shutdown 2024-05-15T13:10:17.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.691+0000 7fcfd2f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:17.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.691+0000 7fcfd2f64640 1 -- 172.21.15.49:0/687919547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 msgr2=0x7fcfcc142cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:17.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.691+0000 7fcfd2f64640 1 --2- 172.21.15.49:0/687919547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 0x7fcfcc142cb0 secure :-1 s=READY pgs=5445 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0b4660 tx=0x7fcfb802b840 comp rx=0 tx=0).stop 2024-05-15T13:10:17.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.691+0000 7fcfd2f64640 1 -- 172.21.15.49:0/687919547 shutdown_connections 2024-05-15T13:10:17.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.691+0000 7fcfd2f64640 1 --2- 172.21.15.49:0/687919547 >> v2:172.21.15.49:3300/0 conn(0x7fcfcc1431f0 0x7fcfcc13f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.691+0000 7fcfd2f64640 1 --2- 172.21.15.49:0/687919547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 0x7fcfcc142cb0 unknown :-1 s=CLOSED pgs=5445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.691+0000 7fcfd2f64640 1 -- 172.21.15.49:0/687919547 >> 172.21.15.49:0/687919547 conn(0x7fcfcc066ed0 msgr2=0x7fcfcc0672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:17.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.691+0000 7fcfd2f64640 1 -- 172.21.15.49:0/687919547 shutdown_connections 2024-05-15T13:10:17.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.691+0000 7fcfd2f64640 1 -- 172.21.15.49:0/687919547 wait complete. 2024-05-15T13:10:17.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.692+0000 7fcfd2f64640 10 monclient: build_initial_monmap 2024-05-15T13:10:17.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.692+0000 7fcfd2f64640 10 monclient: monmap: 2024-05-15T13:10:17.695 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:17.695 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:17.695 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:17.696 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:17.696 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:17.696 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.696 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:17.696 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:17.696 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:17.697 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.692+0000 7fcfd2f64640 1 librados: starting msgr at 2024-05-15T13:10:17.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.692+0000 7fcfd2f64640 1 librados: starting objecter 2024-05-15T13:10:17.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.692+0000 7fcfd2f64640 1 Processor -- start 2024-05-15T13:10:17.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.692+0000 7fcfd2f64640 1 -- start start 2024-05-15T13:10:17.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.692+0000 7fcfd2f64640 1 librados: setting wanted keys 2024-05-15T13:10:17.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.692+0000 7fcfd2f64640 1 librados: calling monclient init 2024-05-15T13:10:17.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.692+0000 7fcfd2f64640 10 monclient: init 2024-05-15T13:10:17.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:17.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T13:10:17.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcfcc0b53e0 0x7fcfcc1db0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 10 monclient(hunting): picked mon.noname-c con 0x7fcfcc0b53e0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:17.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 0x7fcfcc13a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 10 monclient(hunting): picked mon.noname-b con 0x7fcfcc142870 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:17.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfcc1431f0 0x7fcfcc13a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 10 monclient(hunting): picked mon.noname-a con 0x7fcfcc1431f0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:17.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:17.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc13ff30 con 0x7fcfcc1431f0 2024-05-15T13:10:17.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:17.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc13fdb0 con 0x7fcfcc142870 2024-05-15T13:10:17.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:17.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc1400b0 con 0x7fcfcc0b53e0 2024-05-15T13:10:17.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:17.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfd2f64640 10 monclient(hunting): authenticate will time out at 3775.854913s 2024-05-15T13:10:17.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.693+0000 7fcfcbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 0x7fcfcc13a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfd0d00640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcfcc0b53e0 0x7fcfcc1db0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfcbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 0x7fcfcc13a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43076/0 (socket says 172.21.15.49:43076) 2024-05-15T13:10:17.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfcbfff640 1 -- 172.21.15.49:0/1184308094 learned_addr learned my addr 172.21.15.49:0/1184308094 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:17.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfcbfff640 10 monclient(hunting): get_auth_request con 0x7fcfcc142870 auth_method 0 2024-05-15T13:10:17.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:17.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfcbfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfd0d00640 10 monclient(hunting): get_auth_request con 0x7fcfcc0b53e0 auth_method 0 2024-05-15T13:10:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfd0d00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfd0d00640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfd0d00640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:17.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.694+0000 7fcfcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:17.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.695+0000 7fcfd0d00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:17.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.695+0000 7fcfd0d00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:17.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.695+0000 7fcfd0d00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:17.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.697+0000 7fcfd1501640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfcc1431f0 0x7fcfcc13a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfd1501640 10 monclient(hunting): get_auth_request con 0x7fcfcc1431f0 auth_method 0 2024-05-15T13:10:17.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfd1501640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:17.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfd1501640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:17.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfd1501640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:17.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfcbfff640 10 monclient(hunting): handle_auth_done global_id 22000 payload 824 2024-05-15T13:10:17.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfcbfff640 1 -- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcfcc0b53e0 msgr2=0x7fcfcc1db0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:17.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfcbfff640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcfcc0b53e0 0x7fcfcc1db0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfcbfff640 1 -- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfcc1431f0 msgr2=0x7fcfcc13a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:17.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfcbfff640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfcc1431f0 0x7fcfcc13a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:17.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfcbfff640 10 monclient: _finish_hunting 0 2024-05-15T13:10:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfcbfff640 1 monclient: found mon.noname-b 2024-05-15T13:10:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfcbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfcbfff640 1 -- 172.21.15.49:0/1184308094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc1206d0 con 0x7fcfcc142870 2024-05-15T13:10:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.698+0000 7fcfd1501640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfcc1431f0 0x7fcfcc13a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-15T13:10:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.702+0000 7fcfcbfff640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 0x7fcfcc13a440 secure :-1 s=READY pgs=5446 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009890 tx=0x7fcfb80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.702+0000 7fcfc97fa640 10 client.?.objecter ms_handle_connect 0x7fcfcc142870 2024-05-15T13:10:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.702+0000 7fcfc97fa640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 1 -- 172.21.15.49:0/1184308094 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8032840 con 0x7fcfcc142870 2024-05-15T13:10:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 10 monclient: dump: 2024-05-15T13:10:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:17.715 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 10 monclient: _finish_auth 0 2024-05-15T13:10:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 10 monclient: _check_auth_tickets 2024-05-15T13:10:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 1 -- 172.21.15.49:0/1184308094 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb8031ec0 con 0x7fcfcc142870 2024-05-15T13:10:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfd2f64640 5 monclient: authenticate success, global_id 22000 2024-05-15T13:10:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfd2f64640 10 monclient: _renew_subs 2024-05-15T13:10:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfd2f64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfd2f64640 1 -- 172.21.15.49:0/1184308094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc1209b0 con 0x7fcfcc142870 2024-05-15T13:10:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 1 -- 172.21.15.49:0/1184308094 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8032be0 con 0x7fcfcc142870 2024-05-15T13:10:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfc97fa640 10 monclient: dump: 2024-05-15T13:10:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:17.718 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfcaffd640 4 set_mon_vals no callback set 2024-05-15T13:10:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfd2f64640 10 client.22000.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfd2f64640 10 monclient: _renew_subs 2024-05-15T13:10:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfd2f64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfd2f64640 1 -- 172.21.15.49:0/1184308094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc0b0760 con 0x7fcfcc142870 2024-05-15T13:10:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.708+0000 7fcfd2f64640 1 librados: init done 2024-05-15T13:10:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.709+0000 7fcfd2f64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.709+0000 7fcfd2f64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.709+0000 7fcfd2f64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.709+0000 7fcfd2f64640 1 -- 172.21.15.49:0/1184308094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfcc04f0e0 con 0x7fcfcc142870 2024-05-15T13:10:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.728+0000 7fcfc97fa640 1 -- 172.21.15.49:0/1184308094 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcfb80453f0 con 0x7fcfcc142870 2024-05-15T13:10:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.728+0000 7fcfc97fa640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fcfb006ceb0 0x7fcfb006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.728+0000 7fcfd0d00640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fcfb006ceb0 0x7fcfb006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.729+0000 7fcfd0d00640 10 monclient: get_auth_request con 0x7fcfb006ceb0 auth_method 0 2024-05-15T13:10:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.729+0000 7fcfd0d00640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fcfb006ceb0 0x7fcfb006f370 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0007fd0 tx=0x7fcfc0007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:17.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.729+0000 7fcfc97fa640 10 client.22000.objecter ms_handle_connect 0x7fcfb006ceb0 2024-05-15T13:10:17.759 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:46.968 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:17.761 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:46.969 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:17.788 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:46.996 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:17.814 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:47.022 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:17.815 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:47.024 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:17.843 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:47.050 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.959+0000 7fcfc97fa640 1 -- 172.21.15.49:0/1184308094 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fcfb802f030 con 0x7fcfcc142870 2024-05-15T13:10:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.959+0000 7fcfc97fa640 10 client.22000.objecter ms_dispatch 0x7fcfcc000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.959+0000 7fcfc97fa640 3 client.22000.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.959+0000 7fcfc97fa640 3 client.22000.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.959+0000 7fcfc97fa640 20 client.22000.objecter dump_active .. 0 homeless 2024-05-15T13:10:17.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.969+0000 7fcfc97fa640 1 -- 172.21.15.49:0/1184308094 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcfb8040070 con 0x7fcfcc142870 2024-05-15T13:10:17.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.969+0000 7fcfc97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:17.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:17.969+0000 7fcfc97fa640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:18.009 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:47.176 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:18.010 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:47.178 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:18.036 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:47.203 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:18.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.163+0000 7fcfd2f64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-15T13:10:18.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.163+0000 7fcfd2f64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T13:10:18.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.163+0000 7fcfd2f64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:18.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.163+0000 7fcfd2f64640 1 -- 172.21.15.49:0/1184308094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfcc0716f0 con 0x7fcfcc142870 2024-05-15T13:10:18.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.219+0000 7fcfc97fa640 1 -- 172.21.15.49:0/1184308094 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1251) v1 ==== 74+0+19521 (secure 0 0 0) 0x7fcfb807b8c0 con 0x7fcfcc142870 2024-05-15T13:10:18.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.219+0000 7fcfc97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-15T13:10:18.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.219+0000 7fcfc97fa640 10 monclient: _finish_command 2 = system:0 2024-05-15T13:10:18.209 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T13:10:18.210 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1251,"fsid":"8874b84e-6b11-44d3-b531-be645feab39a","created":"2024-05-15T12:31:42.563249+0000","modified":"2024-05-15T13:10:17.579406+0000","last_up_change":"2024-05-15T12:32:41.022082+0000","last_in_change":"2024-05-15T12:31:48.140010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":112,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:32:30.595515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-15T12:32:45.576288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-05-15T12:34:28.609061+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":64,"pool_name":"ceph_test_rados_open_pools_parallel.smithi049-40885","create_time":"2024-05-15T12:35:13.531683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a595cec4-7a89-4922-b2cf-adbd47129fd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6819","nonce":3376151072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6821","nonce":3376151072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6825","nonce":3376151072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3376151072},{"type":"v1","addr":"172.21.15.49:6823","nonce":3376151072}]},"public_addr":"172.21.15.49:6819/3376151072","cluster_addr":"172.21.15.49:6821/3376151072","heartbeat_back_addr":"172.21.15.49:6825/3376151072","heartbeat_front_addr":"172.21.15.49:6823/3376151072","state":["exists","up"]},{"osd":1,"uuid":"f61e8aee-68b5-4786-a307-4865e529e0cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":1232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6811","nonce":3250791401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6813","nonce":3250791401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6817","nonce":3250791401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3250791401},{"type":"v1","addr":"172.21.15.49:6815","nonce":3250791401}]},"public_addr":"172.21.15.49:6811/3250791401","cluster_addr":"172.21.15.49:6813/3250791401","heartbeat_back_addr":"172.21.15.49:6817/3250791401","heartbeat_front_addr":"172.21.15.49:6815/3250791401","state":["exists","up"]},{"osd":2,"uuid":"e3f252e2-0a0b-41a6-83be-fece689b794e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":1239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6827","nonce":2361777220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6829","nonce":2361777220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6833","nonce":2361777220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2361777220},{"type":"v1","addr":"172.21.15.49:6831","nonce":2361777220}]},"public_addr":"172.21.15.49:6827/2361777220","cluster_addr":"172.21.15.49:6829/2361777220","heartbeat_back_addr":"172.21.15.49:6833/2361777220","heartbeat_front_addr":"172.21.15.49:6831/2361777220","state":["exists","up"]},{"osd":3,"uuid":"1019eca4-3c10-4fcd-b8d6-b9dfdd91ed9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6803","nonce":945312262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6805","nonce":945312262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6809","nonce":945312262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":945312262},{"type":"v1","addr":"172.21.15.49:6807","nonce":945312262}]},"public_addr":"172.21.15.49:6803/945312262","cluster_addr":"172.21.15.49:6805/945312262","heartbeat_back_addr":"172.21.15.49:6809/945312262","heartbeat_front_addr":"172.21.15.49:6807/945312262","state":["exists","up"]},{"osd":4,"uuid":"9be5d412-2421-4c86-8d81-540b24dc2616","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6817","nonce":3384996234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6819","nonce":3384996234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6823","nonce":3384996234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3384996234},{"type":"v1","addr":"172.21.15.88:6821","nonce":3384996234}]},"public_addr":"172.21.15.88:6817/3384996234","cluster_addr":"172.21.15.88:6819/3384996234","heartbeat_back_addr":"172.21.15.88:6823/3384996234","heartbeat_front_addr":"172.21.15.88:6821/3384996234","state":["exists","up"]},{"osd":5,"uuid":"d05e6dcc-742b-448b-853f-375b06280a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6809","nonce":3085185190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6811","nonce":3085185190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6815","nonce":3085185190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3085185190},{"type":"v1","addr":"172.21.15.88:6813","nonce":3085185190}]},"public_addr":"172.21.15.88:6809/3085185190","cluster_addr":"172.21.15.88:6811/3085185190","heartbeat_back_addr":"172.21.15.88:6815/3085185190","heartbeat_front_addr":"172.21.15.88:6813/3085185190","state":["exists","up"]},{"osd":6,"uuid":"341b7da5-8516-462d-b852-e867a5dd5e95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6801","nonce":3138490191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6803","nonce":3138490191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6807","nonce":3138490191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3138490191},{"type":"v1","addr":"172.21.15.88:6805","nonce":3138490191}]},"public_addr":"172.21.15.88:6801/3138490191","cluster_addr":"172.21.15.88:6803/3138490191","heartbeat_back_addr":"172.21.15.88:6807/3138490191","heartbeat_front_addr":"172.21.15.88:6805/3138490191","state":["exists","up"]},{"osd":7,"uuid":"ca94bfcf-3295-4f81-a69e-ba59dd3720c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6825","nonce":2037618931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6827","nonce":2037618931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6831","nonce":2037618931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2037618931},{"type":"v1","addr":"172.21.15.88:6829","nonce":2037618931}]},"public_addr":"172.21.15.88:6825/2037618931","cluster_addr":"172.21.15.88:6827/2037618931","heartbeat_back_addr":"172.21.15.88:6831/2037618931","heartbeat_front_addr":"172.21.15.88:6829/2037618931","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:28.806234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:29.001705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:28.402190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:27.582000+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:25.805322+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.269314+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.001531+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:32:26.945037+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":13,"snaps":[{"begin":3,"length":2}]},{"pool":21,"snaps":[{"begin":3,"length":4}]},{"pool":33,"snaps":[{"begin":1,"length":1}]},{"pool":62,"snaps":[{"begin":1,"length":1}]},{"pool":99,"snaps":[{"begin":1,"length":1}]},{"pool":108,"snaps":[{"begin":1,"length":2}]},{"pool":111,"snaps":[{"begin":2,"length":4},{"begin":8,"length":2}]},{"pool":113,"snaps":[{"begin":3,"length":1}]},{"pool":116,"snaps":[{"begin":3,"length":2}]},{"pool":119,"snaps":[{"begin":2,"length":2}]},{"pool":126,"snaps":[{"begin":2,"length":3}]},{"pool":131,"snaps":[{"begin":4,"length":2}]},{"pool":133,"snaps":[{"begin":2,"length":2}]},{"pool":135,"snaps":[{"begin":2,"length":1}]},{"pool":138,"snaps":[{"begin":2,"length":3}]},{"pool":147,"snaps":[{"begin":3,"length":3}]},{"pool":148,"snaps":[{"begin":4,"length":4}]},{"pool":157,"snaps":[{"begin":7,"length":1}]},{"pool":161,"snaps":[{"begin":8,"length":1}]},{"pool":163,"snaps":[{"begin":10,"length":1}]},{"pool":164,"snaps":[{"begin":11,"length":2}]},{"pool":167,"snaps":[{"begin":14,"length":1}]},{"pool":210,"snaps":[{"begin":2,"length":1}]},{"pool":211,"snaps":[{"begin":3,"length":1}]},{"pool":214,"snaps":[{"begin":4,"length":1}]},{"pool":218,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T13:10:18.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 10 librados: watch_flush enter 2024-05-15T13:10:18.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 10 librados: watch_flush exit 2024-05-15T13:10:18.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 20 client.22000.objecter shutdown clearing up homeless session... 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 10 client.22000.objecter successfully canceled tick 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 -- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fcfb006ceb0 msgr2=0x7fcfb006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fcfb006ceb0 0x7fcfb006f370 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0007fd0 tx=0x7fcfc0007e80 comp rx=0 tx=0).stop 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 10 monclient: shutdown 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 -- 172.21.15.49:0/1184308094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 msgr2=0x7fcfcc13a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 0x7fcfcc13a440 secure :-1 s=READY pgs=5446 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009890 tx=0x7fcfb80046c0 comp rx=0 tx=0).stop 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 -- 172.21.15.49:0/1184308094 shutdown_connections 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfcc1431f0 0x7fcfcc13a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fcfb006ceb0 0x7fcfb006f370 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc142870 0x7fcfcc13a440 unknown :-1 s=CLOSED pgs=5446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 --2- 172.21.15.49:0/1184308094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcfcc0b53e0 0x7fcfcc1db0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 -- 172.21.15.49:0/1184308094 >> 172.21.15.49:0/1184308094 conn(0x7fcfcc066ed0 msgr2=0x7fcfcc0672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 -- 172.21.15.49:0/1184308094 shutdown_connections 2024-05-15T13:10:18.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 -- 172.21.15.49:0/1184308094 wait complete. 2024-05-15T13:10:18.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:18.222+0000 7fcfd2f64640 1 librados: shutdown 2024-05-15T13:10:19.225 INFO:tasks.ceph:Scrubbing osd.0 2024-05-15T13:10:19.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:10:19.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.346+0000 7fb1d82fa640 10 monclient: get_monmap_and_config 2024-05-15T13:10:19.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.346+0000 7fb1d82fa640 10 monclient: build_initial_monmap 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.346+0000 7fb1d82fa640 10 monclient: monmap: 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:19.346618+0000 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:19.346618+0000 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:19.336 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.346+0000 7fb1d82fa640 1 Processor -- start 2024-05-15T13:10:19.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.346+0000 7fb1d82fa640 1 -- start start 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.346+0000 7fb1d82fa640 10 monclient: init 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d00b5410 0x7fb1d0145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb1d00b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1d00aa0f0 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb1d014af50 0x7fb1d014d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb1d014af50 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d00ab220 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d00b57f0 con 0x7fb1d014af50 2024-05-15T13:10:19.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00a9810 con 0x7fb1d00b5410 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.347+0000 7fb1d82fa640 10 monclient(hunting): authenticate will time out at 3777.508999s 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.348+0000 7fb1d6096640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d00b5410 0x7fb1d0145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.348+0000 7fb1d6096640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d00b5410 0x7fb1d0145bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:37088/0 (socket says 172.21.15.49:37088) 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.348+0000 7fb1d6096640 1 -- 172.21.15.49:0/656809148 learned_addr learned my addr 172.21.15.49:0/656809148 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.348+0000 7fb1d6096640 10 monclient(hunting): get_auth_request con 0x7fb1d00b5410 auth_method 0 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.348+0000 7fb1d6096640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.348+0000 7fb1d6096640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.348+0000 7fb1d6096640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.348+0000 7fb1d6897640 1 --2- 172.21.15.49:0/656809148 >> v2:172.21.15.88:3300/0 conn(0x7fb1d014af50 0x7fb1d014d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.349+0000 7fb1d6096640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:19.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.349+0000 7fb1d6096640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.349+0000 7fb1d6096640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.349+0000 7fb1d6897640 10 monclient(hunting): get_auth_request con 0x7fb1d014af50 auth_method 0 2024-05-15T13:10:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.349+0000 7fb1d6897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.349+0000 7fb1d6897640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:19.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.349+0000 7fb1d6897640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.349+0000 7fb1d6897640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.349+0000 7fb1d6897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.349+0000 7fb1d6897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.350+0000 7fb1d6096640 10 monclient(hunting): handle_auth_done global_id 17606 payload 274 2024-05-15T13:10:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.350+0000 7fb1d6096640 1 -- 172.21.15.49:0/656809148 >> v2:172.21.15.88:3300/0 conn(0x7fb1d014af50 msgr2=0x7fb1d014d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.350+0000 7fb1d6096640 1 --2- 172.21.15.49:0/656809148 >> v2:172.21.15.88:3300/0 conn(0x7fb1d014af50 0x7fb1d014d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.350+0000 7fb1d6096640 1 -- 172.21.15.49:0/656809148 >> v1:172.21.15.88:6789/0 conn(0x7fb1d00aa0f0 legacy=0x7fb1d013f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.350+0000 7fb1d6096640 10 monclient: _finish_hunting 0 2024-05-15T13:10:19.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.350+0000 7fb1d6096640 1 monclient: found mon.noname-b 2024-05-15T13:10:19.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.350+0000 7fb1d6096640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:19.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.350+0000 7fb1d6096640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T13:10:19.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.350+0000 7fb1d6096640 1 -- 172.21.15.49:0/656809148 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d0146110 con 0x7fb1d00b5410 2024-05-15T13:10:19.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.352+0000 7fb1d6096640 1 --2- 172.21.15.49:0/656809148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d00b5410 0x7fb1d0145bd0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0098a0 tx=0x7fb1cc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bbbe5bf0ae27a7e server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 1 -- 172.21.15.49:0/656809148 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc035020 con 0x7fb1d00b5410 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 10 monclient: dump: 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:19.346 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 10 monclient: _finish_auth 0 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 10 monclient: _check_auth_tickets 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 1 -- 172.21.15.49:0/656809148 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1cc031640 con 0x7fb1d00b5410 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d82fa640 5 monclient: authenticate success, global_id 17606 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d82fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 1 -- 172.21.15.49:0/656809148 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc02d040 con 0x7fb1d00b5410 2024-05-15T13:10:19.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d82fa640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:19.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:19.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d82fa640 4 set_mon_vals no callback set 2024-05-15T13:10:19.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:19.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.356+0000 7fb1d5094640 10 monclient: dump: 2024-05-15T13:10:19.349 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:19.349 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:19.349 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.349 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.349 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:19.349 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.349 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:19.349 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:19.350 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:19.350 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 10 monclient: shutdown 2024-05-15T13:10:19.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 1 -- 172.21.15.49:0/656809148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d00b5410 msgr2=0x7fb1d0145bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 1 --2- 172.21.15.49:0/656809148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d00b5410 0x7fb1d0145bd0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0098a0 tx=0x7fb1cc02b8f0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 1 -- 172.21.15.49:0/656809148 shutdown_connections 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 1 --2- 172.21.15.49:0/656809148 >> v2:172.21.15.88:3300/0 conn(0x7fb1d014af50 0x7fb1d014d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 1 --2- 172.21.15.49:0/656809148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d00b5410 0x7fb1d0145bd0 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 1 -- 172.21.15.49:0/656809148 >> 172.21.15.49:0/656809148 conn(0x7fb1d0066f00 msgr2=0x7fb1d0067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 1 -- 172.21.15.49:0/656809148 shutdown_connections 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 1 -- 172.21.15.49:0/656809148 wait complete. 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.357+0000 7fb1d82fa640 10 monclient: build_initial_monmap 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.358+0000 7fb1d82fa640 10 monclient: monmap: 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:19.354 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.358+0000 7fb1d82fa640 1 librados: starting msgr at 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.358+0000 7fb1d82fa640 1 librados: starting objecter 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.358+0000 7fb1d82fa640 1 Processor -- start 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.358+0000 7fb1d82fa640 1 -- start start 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.358+0000 7fb1d82fa640 1 librados: setting wanted keys 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.358+0000 7fb1d82fa640 1 librados: calling monclient init 2024-05-15T13:10:19.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.358+0000 7fb1d82fa640 10 monclient: init 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d00aa0f0 0x7fb1d01373a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb1d00aa0f0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1d00b5410 0x7fb1d0135330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb1d00b5410 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d014af50 0x7fb1d0135870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb1d014af50 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00a9bd0 con 0x7fb1d00b5410 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00ab770 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00aa530 con 0x7fb1d014af50 2024-05-15T13:10:19.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d82fa640 10 monclient(hunting): authenticate will time out at 3777.520555s 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6897640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d014af50 0x7fb1d0135870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6096640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d00aa0f0 0x7fb1d01373a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d5895640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1d00b5410 0x7fb1d0135330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6096640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d00aa0f0 0x7fb1d01373a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43100/0 (socket says 172.21.15.49:43100) 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6096640 1 -- 172.21.15.49:0/2875764164 learned_addr learned my addr 172.21.15.49:0/2875764164 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6096640 10 monclient(hunting): get_auth_request con 0x7fb1d00aa0f0 auth_method 0 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6096640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6096640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6096640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d5895640 10 monclient(hunting): get_auth_request con 0x7fb1d00b5410 auth_method 0 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d5895640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d5895640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:19.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d5895640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6897640 10 monclient(hunting): get_auth_request con 0x7fb1d014af50 auth_method 0 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6897640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.359+0000 7fb1d6897640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.360+0000 7fb1d6096640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.360+0000 7fb1d6096640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.360+0000 7fb1d6096640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.360+0000 7fb1d6897640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.360+0000 7fb1d6897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.360+0000 7fb1d6897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.360+0000 7fb1d5895640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.360+0000 7fb1d5895640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.360+0000 7fb1d5895640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:19.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.363+0000 7fb1d6096640 10 monclient(hunting): handle_auth_done global_id 22009 payload 824 2024-05-15T13:10:19.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.363+0000 7fb1d6096640 1 -- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d014af50 msgr2=0x7fb1d0135870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.363+0000 7fb1d6096640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1d014af50 0x7fb1d0135870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.363+0000 7fb1d6096640 1 -- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1d00b5410 msgr2=0x7fb1d0135330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.363+0000 7fb1d6096640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1d00b5410 0x7fb1d0135330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.363+0000 7fb1d6096640 10 monclient: _finish_hunting 0 2024-05-15T13:10:19.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.363+0000 7fb1d6096640 1 monclient: found mon.noname-b 2024-05-15T13:10:19.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.363+0000 7fb1d6096640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:19.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.363+0000 7fb1d6096640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:19.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.363+0000 7fb1d6096640 1 -- 172.21.15.49:0/2875764164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d0140950 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.365+0000 7fb1d6096640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d00aa0f0 0x7fb1d01373a0 secure :-1 s=READY pgs=5448 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc000c00 tx=0x7fb1cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.365+0000 7fb1c6ffd640 10 client.?.objecter ms_handle_connect 0x7fb1d00aa0f0 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.365+0000 7fb1c6ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 1 -- 172.21.15.49:0/2875764164 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc040070 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 10 monclient: dump: 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:19.361 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 10 monclient: _finish_auth 0 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 10 monclient: _check_auth_tickets 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 1 -- 172.21.15.49:0/2875764164 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1cc0399f0 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 1 -- 172.21.15.49:0/2875764164 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc035040 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1d82fa640 5 monclient: authenticate success, global_id 22009 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1d82fa640 10 monclient: _renew_subs 2024-05-15T13:10:19.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1d82fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d0135db0 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1c6ffd640 10 monclient: dump: 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.368+0000 7fb1d4893640 4 set_mon_vals no callback set 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.369+0000 7fb1d82fa640 10 client.22009.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.369+0000 7fb1d82fa640 10 monclient: _renew_subs 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.369+0000 7fb1d82fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.369+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d0145a90 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.369+0000 7fb1d82fa640 1 librados: init done 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.369+0000 7fb1d82fa640 10 client.22009.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.369+0000 7fb1d82fa640 15 client.22009.objecter _session_command_op_assign -1 1 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.370+0000 7fb1d82fa640 20 client.22009.objecter _get_session osd=-1 returning homeless 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.370+0000 7fb1d82fa640 10 client.22009.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.370+0000 7fb1d82fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:19.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.370+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb1d004f0e0 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.388+0000 7fb1c6ffd640 1 -- 172.21.15.49:0/2875764164 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb1cc038020 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.389+0000 7fb1c6ffd640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb1b406ceb0 0x7fb1b406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.389+0000 7fb1d5895640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb1b406ceb0 0x7fb1b406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.389+0000 7fb1d5895640 10 monclient: get_auth_request con 0x7fb1b406ceb0 auth_method 0 2024-05-15T13:10:19.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.389+0000 7fb1c6ffd640 10 client.22009.objecter ms_handle_connect 0x7fb1b406ceb0 2024-05-15T13:10:19.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.389+0000 7fb1d5895640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb1b406ceb0 0x7fb1b406f370 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0137100 tx=0x7fb1c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 1 -- 172.21.15.49:0/2875764164 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fb1cc078e20 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 10 client.22009.objecter ms_dispatch 0x7fb1d0000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 3 client.22009.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 3 client.22009.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 10 client.22009.objecter checking command 1 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] conn(0x7fb1b40758a0 0x7fb1b4077d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 20 client.22009.objecter _get_session s=0x7fb1b40751d0 osd=0 3 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 20 client.22009.objecter put_session s=0x7fb1b40751d0 osd=0 3 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 15 client.22009.objecter _session_command_op_remove -1 1 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 20 client.22009.objecter _get_session s=0x7fb1b40751d0 osd=0 3 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 20 client.22009.objecter get_session s=0x7fb1b40751d0 osd=0 3 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 15 client.22009.objecter _session_command_op_assign 0 1 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 20 client.22009.objecter put_session s=0x7fb1b40751d0 osd=0 4 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 10 client.22009.objecter _send_command 1 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 1 -- 172.21.15.49:0/2875764164 --> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1cc034e90 con 0x7fb1b40758a0 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 20 client.22009.objecter dump_active .. 0 homeless 2024-05-15T13:10:19.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 1 -- 172.21.15.49:0/2875764164 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=1251) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1cc077020 con 0x7fb1d00aa0f0 2024-05-15T13:10:19.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.392+0000 7fb1c6ffd640 10 monclient: handle_get_version_reply finishing 1 version 1251 2024-05-15T13:10:19.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.393+0000 7fb1d6897640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] conn(0x7fb1b40758a0 0x7fb1b4077d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.393+0000 7fb1d6897640 10 monclient: get_auth_request con 0x7fb1b40758a0 auth_method 0 2024-05-15T13:10:19.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.396+0000 7fb1d6897640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] conn(0x7fb1b40758a0 0x7fb1b4077d40 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:19.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.396+0000 7fb1c6ffd640 10 client.22009.objecter ms_handle_connect 0x7fb1b40758a0 2024-05-15T13:10:19.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.422+0000 7fb1c6ffd640 1 -- 172.21.15.49:0/2875764164 <== osd.0 v2:172.21.15.49:6818/3376151072 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb1cc034e90 con 0x7fb1b40758a0 2024-05-15T13:10:19.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.422+0000 7fb1c6ffd640 10 client.22009.objecter ms_dispatch 0x7fb1d0000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:10:19.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.422+0000 7fb1c6ffd640 10 client.22009.objecter _finish_command 1 = system:0 2024-05-15T13:10:19.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.422+0000 7fb1c6ffd640 20 client.22009.objecter put_session s=0x7fb1b40751d0 osd=0 4 2024-05-15T13:10:19.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.422+0000 7fb1c6ffd640 15 client.22009.objecter _session_command_op_remove 0 1 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 10 client.22009.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 15 client.22009.objecter _session_command_op_assign -1 2 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 20 client.22009.objecter _get_session s=0x7fb1b40751d0 osd=0 3 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 20 client.22009.objecter put_session s=0x7fb1b40751d0 osd=0 3 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 20 client.22009.objecter _get_session s=0x7fb1b40751d0 osd=0 3 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 15 client.22009.objecter _session_command_op_remove -1 2 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 20 client.22009.objecter get_session s=0x7fb1b40751d0 osd=0 3 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 15 client.22009.objecter _session_command_op_assign 0 2 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 20 client.22009.objecter put_session s=0x7fb1b40751d0 osd=0 4 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 10 client.22009.objecter _send_command 2 2024-05-15T13:10:19.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.435+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 --> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb1d0140730 con 0x7fb1b40758a0 2024-05-15T13:10:19.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1c6ffd640 1 -- 172.21.15.49:0/2875764164 <== osd.0 v2:172.21.15.49:6818/3376151072 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb1d0140730 con 0x7fb1b40758a0 2024-05-15T13:10:19.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1c6ffd640 10 client.22009.objecter ms_dispatch 0x7fb1d0000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:10:19.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1c6ffd640 10 client.22009.objecter _finish_command 2 = system:0 2024-05-15T13:10:19.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1c6ffd640 20 client.22009.objecter put_session s=0x7fb1b40751d0 osd=0 4 2024-05-15T13:10:19.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1c6ffd640 15 client.22009.objecter _session_command_op_remove 0 2 2024-05-15T13:10:19.439 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-05-15T13:10:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:10:19.439 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-05-15T13:10:19.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1d82fa640 10 librados: watch_flush enter 2024-05-15T13:10:19.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1d82fa640 10 librados: watch_flush exit 2024-05-15T13:10:19.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1d82fa640 10 client.22009.objecter close_session for osd.0 2024-05-15T13:10:19.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] conn(0x7fb1b40758a0 msgr2=0x7fb1b4077d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1d82fa640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:6818/3376151072,v1:172.21.15.49:6819/3376151072] conn(0x7fb1b40758a0 0x7fb1b4077d40 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1d82fa640 20 client.22009.objecter put_session s=0x7fb1b40751d0 osd=0 1 2024-05-15T13:10:19.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1d82fa640 20 client.22009.objecter shutdown clearing up homeless session... 2024-05-15T13:10:19.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.449+0000 7fb1d82fa640 10 client.22009.objecter successfully canceled tick 2024-05-15T13:10:19.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb1b406ceb0 msgr2=0x7fb1b406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fb1b406ceb0 0x7fb1b406f370 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0137100 tx=0x7fb1c0009040 comp rx=0 tx=0).stop 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 10 monclient: shutdown 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d00aa0f0 msgr2=0x7fb1d01373a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 1 --2- 172.21.15.49:0/2875764164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d00aa0f0 0x7fb1d01373a0 secure :-1 s=READY pgs=5448 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc000c00 tx=0x7fb1cc0046c0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d6897640 1 -- 172.21.15.49:0/2875764164 reap_dead start 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 shutdown_connections 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 >> 172.21.15.49:0/2875764164 conn(0x7fb1d0066f00 msgr2=0x7fb1d00b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 shutdown_connections 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 1 -- 172.21.15.49:0/2875764164 wait complete. 2024-05-15T13:10:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.450+0000 7fb1d82fa640 1 librados: shutdown 2024-05-15T13:10:19.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-15T13:10:19.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.568+0000 7f97134fa640 10 monclient: get_monmap_and_config 2024-05-15T13:10:19.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.568+0000 7f97134fa640 10 monclient: build_initial_monmap 2024-05-15T13:10:19.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.568+0000 7f97134fa640 10 monclient: monmap: 2024-05-15T13:10:19.558 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:19.558 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:19.558 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:19.569135+0000 2024-05-15T13:10:19.558 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:19.569135+0000 2024-05-15T13:10:19.558 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:19.558 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.559 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:19.559 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:19.559 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:19.559 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:19.559 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:19.559 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.569+0000 7f97134fa640 1 Processor -- start 2024-05-15T13:10:19.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.569+0000 7f97134fa640 1 -- start start 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.569+0000 7f97134fa640 10 monclient: init 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f970c0b53e0 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f970c142870 0x7f970c142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 10 monclient(hunting): picked mon.noname-b con 0x7f970c142870 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f970c1431f0 0x7f970c13f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 10 monclient(hunting): picked mon.noname-a con 0x7f970c1431f0 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c0ab230 con 0x7f970c0b53e0 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970c0b57c0 con 0x7f970c1431f0 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970c0aa230 con 0x7f970c142870 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.570+0000 7f97134fa640 10 monclient(hunting): authenticate will time out at 3777.731616s 2024-05-15T13:10:19.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.571+0000 7f9711a97640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f970c1431f0 0x7f970c13f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.571+0000 7f9711296640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f970c0b53e0 0x7f970c0a99f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:55244/0 (socket says 172.21.15.49:55244) 2024-05-15T13:10:19.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.571+0000 7f9711296640 1 -- 172.21.15.49:0/4074107194 learned_addr learned my addr 172.21.15.49:0/4074107194 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:19.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.571+0000 7f9711a97640 10 monclient(hunting): get_auth_request con 0x7f970c1431f0 auth_method 0 2024-05-15T13:10:19.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.571+0000 7f9711a97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:19.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.571+0000 7f9711a97640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:19.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.571+0000 7f9711a97640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:19.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.572+0000 7f9711a97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:19.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.572+0000 7f9711a97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:19.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.572+0000 7f9711a97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:19.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.575+0000 7f9711a97640 10 monclient(hunting): handle_auth_done global_id 16167 payload 274 2024-05-15T13:10:19.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.575+0000 7f9711a97640 1 -- 172.21.15.49:0/4074107194 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f970c142870 msgr2=0x7f970c142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:19.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.575+0000 7f9711a97640 1 --2- 172.21.15.49:0/4074107194 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f970c142870 0x7f970c142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.575+0000 7f9711a97640 1 -- 172.21.15.49:0/4074107194 >> v1:172.21.15.49:6789/0 conn(0x7f970c0b53e0 legacy=0x7f970c0a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.575+0000 7f9711a97640 10 monclient: _finish_hunting 0 2024-05-15T13:10:19.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.575+0000 7f9711a97640 1 monclient: found mon.noname-a 2024-05-15T13:10:19.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.575+0000 7f9711a97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:19.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.575+0000 7f9711a97640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T13:10:19.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.575+0000 7f9711a97640 1 -- 172.21.15.49:0/4074107194 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c0ab510 con 0x7f970c1431f0 2024-05-15T13:10:19.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.579+0000 7f9711a97640 1 --2- 172.21.15.49:0/4074107194 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f970c1431f0 0x7f970c13f660 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f970c0aa3b0 tx=0x7f970402be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efdc7a09283db74 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:19.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 1 -- 172.21.15.49:0/4074107194 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704039070 con 0x7f970c1431f0 2024-05-15T13:10:19.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:19.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:10:19.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 10 monclient: dump: 2024-05-15T13:10:19.573 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:19.573 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:19.573 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.573 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.573 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 10 monclient: _finish_auth 0 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 10 monclient: _check_auth_tickets 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 1 -- 172.21.15.49:0/4074107194 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970402cbb0 con 0x7f970c1431f0 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 1 -- 172.21.15.49:0/4074107194 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704034440 con 0x7f970c1431f0 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f97134fa640 5 monclient: authenticate success, global_id 16167 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f97134fa640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f97134fa640 4 set_mon_vals no callback set 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.583+0000 7f9703fff640 10 monclient: dump: 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:19.575 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.584+0000 7f97134fa640 10 monclient: shutdown 2024-05-15T13:10:19.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.584+0000 7f97134fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.584+0000 7f97134fa640 1 -- 172.21.15.49:0/4074107194 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f970c1431f0 msgr2=0x7f970c13f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.584+0000 7f97134fa640 1 --2- 172.21.15.49:0/4074107194 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f970c1431f0 0x7f970c13f660 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f970c0aa3b0 tx=0x7f970402be00 comp rx=0 tx=0).stop 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.584+0000 7f97134fa640 1 -- 172.21.15.49:0/4074107194 shutdown_connections 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.584+0000 7f97134fa640 1 --2- 172.21.15.49:0/4074107194 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f970c1431f0 0x7f970c13f660 unknown :-1 s=CLOSED pgs=3260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.584+0000 7f97134fa640 1 --2- 172.21.15.49:0/4074107194 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f970c142870 0x7f970c142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.584+0000 7f97134fa640 1 -- 172.21.15.49:0/4074107194 >> 172.21.15.49:0/4074107194 conn(0x7f970c066ed0 msgr2=0x7f970c0672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.584+0000 7f97134fa640 1 -- 172.21.15.49:0/4074107194 shutdown_connections 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.584+0000 7f97134fa640 1 -- 172.21.15.49:0/4074107194 wait complete. 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.585+0000 7f97134fa640 10 monclient: build_initial_monmap 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.585+0000 7f97134fa640 10 monclient: monmap: 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.586+0000 7f97134fa640 1 librados: starting msgr at 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.586+0000 7f97134fa640 1 librados: starting objecter 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.586+0000 7f97134fa640 1 Processor -- start 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.586+0000 7f97134fa640 1 -- start start 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.586+0000 7f97134fa640 1 librados: setting wanted keys 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.586+0000 7f97134fa640 1 librados: calling monclient init 2024-05-15T13:10:19.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.586+0000 7f97134fa640 10 monclient: init 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f970c0b53e0 0x7f970c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 10 monclient(hunting): picked mon.noname-a con 0x7f970c0b53e0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970c142870 0x7f970c1db370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 10 monclient(hunting): picked mon.noname-b con 0x7f970c142870 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f970c1431f0 0x7f970c139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 10 monclient(hunting): picked mon.noname-c con 0x7f970c1431f0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c13ff30 con 0x7f970c0b53e0 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c13fdb0 con 0x7f970c142870 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970c1400b0 con 0x7f970c1431f0 2024-05-15T13:10:19.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f97134fa640 10 monclient(hunting): authenticate will time out at 3777.748878s 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.587+0000 7f9711296640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f970c0b53e0 0x7f970c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9710a95640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970c142870 0x7f970c1db370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9711296640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f970c0b53e0 0x7f970c13a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59922/0 (socket says 172.21.15.49:59922) 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9711296640 1 -- 172.21.15.49:0/3629753070 learned_addr learned my addr 172.21.15.49:0/3629753070 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9711296640 10 monclient(hunting): get_auth_request con 0x7f970c0b53e0 auth_method 0 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9711296640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9711296640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9711296640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9710a95640 10 monclient(hunting): get_auth_request con 0x7f970c142870 auth_method 0 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9710a95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9710a95640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9710a95640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:19.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9711296640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:19.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9711296640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:19.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9711296640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:19.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9710a95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:19.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9710a95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:19.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.588+0000 7f9710a95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.591+0000 7f9710a95640 10 monclient(hunting): handle_auth_done global_id 22012 payload 824 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.591+0000 7f9710a95640 1 -- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f970c1431f0 msgr2=0x7f970c139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.591+0000 7f9710a95640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f970c1431f0 0x7f970c139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.591+0000 7f9710a95640 1 -- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f970c0b53e0 msgr2=0x7f970c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.591+0000 7f9710a95640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f970c0b53e0 0x7f970c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.591+0000 7f9710a95640 10 monclient: _finish_hunting 0 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.591+0000 7f9710a95640 1 monclient: found mon.noname-b 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.591+0000 7f9710a95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.591+0000 7f9710a95640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.591+0000 7f9710a95640 1 -- 172.21.15.49:0/3629753070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c13ae50 con 0x7f970c142870 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.593+0000 7f9710a95640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970c142870 0x7f970c1db370 secure :-1 s=READY pgs=5449 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00b3f0 tx=0x7f96fc00b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.593+0000 7f9701ffb640 10 client.?.objecter ms_handle_connect 0x7f970c142870 2024-05-15T13:10:19.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.593+0000 7f9701ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 1 -- 172.21.15.49:0/3629753070 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc015020 con 0x7f970c142870 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 10 monclient: dump: 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 10 monclient: _finish_auth 0 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 10 monclient: _check_auth_tickets 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 1 -- 172.21.15.49:0/3629753070 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc011830 con 0x7f970c142870 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f97134fa640 5 monclient: authenticate success, global_id 22012 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f97134fa640 10 monclient: _renew_subs 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f97134fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f97134fa640 1 -- 172.21.15.49:0/3629753070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970c1db8b0 con 0x7f970c142870 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 1 -- 172.21.15.49:0/3629753070 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc011c20 con 0x7f970c142870 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f9701ffb640 10 monclient: dump: 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:19.588 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f97037fe640 4 set_mon_vals no callback set 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f97134fa640 10 client.22012.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f97134fa640 10 monclient: _renew_subs 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f97134fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f97134fa640 1 -- 172.21.15.49:0/3629753070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970c139e70 con 0x7f970c142870 2024-05-15T13:10:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.597+0000 7f97134fa640 1 librados: init done 2024-05-15T13:10:19.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.598+0000 7f97134fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:19.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.598+0000 7f97134fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:19.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.598+0000 7f97134fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:19.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.598+0000 7f97134fa640 1 -- 172.21.15.49:0/3629753070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970c04f0e0 con 0x7f970c142870 2024-05-15T13:10:19.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.838+0000 7f9701ffb640 1 -- 172.21.15.49:0/3629753070 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f96fc018020 con 0x7f970c142870 2024-05-15T13:10:19.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.839+0000 7f9701ffb640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f96f006ce60 0x7f96f006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:19.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.839+0000 7f9711296640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f96f006ce60 0x7f96f006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:19.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.839+0000 7f9711296640 10 monclient: get_auth_request con 0x7f96f006ce60 auth_method 0 2024-05-15T13:10:19.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.839+0000 7f9711296640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f96f006ce60 0x7f96f006f320 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9708004e40 tx=0x7f9708007f10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:19.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.839+0000 7f9701ffb640 10 client.22012.objecter ms_handle_connect 0x7f96f006ce60 2024-05-15T13:10:19.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.842+0000 7f9701ffb640 1 -- 172.21.15.49:0/3629753070 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f96fc056df0 con 0x7f970c142870 2024-05-15T13:10:19.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.842+0000 7f9701ffb640 10 client.22012.objecter ms_dispatch 0x7f970c000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:19.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.842+0000 7f9701ffb640 3 client.22012.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:19.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.843+0000 7f9701ffb640 3 client.22012.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:19.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.843+0000 7f9701ffb640 20 client.22012.objecter dump_active .. 0 homeless 2024-05-15T13:10:19.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.852+0000 7f9701ffb640 1 -- 172.21.15.49:0/3629753070 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f96fc01f020 con 0x7f970c142870 2024-05-15T13:10:19.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.852+0000 7f9701ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:19.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:19.852+0000 7f9701ffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:20.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.033+0000 7f97134fa640 1 -- 172.21.15.49:0/3629753070 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f970c1dbc40 con 0x7f96f006ce60 2024-05-15T13:10:20.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.034+0000 7f9701ffb640 1 -- 172.21.15.49:0/3629753070 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f970c1dbc40 con 0x7f96f006ce60 2024-05-15T13:10:20.023 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 0 to deep-scrub 2024-05-15T13:10:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 10 librados: watch_flush enter 2024-05-15T13:10:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 10 librados: watch_flush exit 2024-05-15T13:10:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 20 client.22012.objecter shutdown clearing up homeless session... 2024-05-15T13:10:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 10 client.22012.objecter successfully canceled tick 2024-05-15T13:10:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 -- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f96f006ce60 msgr2=0x7f96f006f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f96f006ce60 0x7f96f006f320 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9708004e40 tx=0x7f9708007f10 comp rx=0 tx=0).stop 2024-05-15T13:10:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 10 monclient: shutdown 2024-05-15T13:10:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 -- 172.21.15.49:0/3629753070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970c142870 msgr2=0x7f970c1db370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970c142870 0x7f970c1db370 secure :-1 s=READY pgs=5449 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00b3f0 tx=0x7f96fc00b8b0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 -- 172.21.15.49:0/3629753070 shutdown_connections 2024-05-15T13:10:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f970c1431f0 0x7f970c139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f96f006ce60 0x7f96f006f320 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f970c142870 0x7f970c1db370 unknown :-1 s=CLOSED pgs=5449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 --2- 172.21.15.49:0/3629753070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f970c0b53e0 0x7f970c13a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 -- 172.21.15.49:0/3629753070 >> 172.21.15.49:0/3629753070 conn(0x7f970c066ed0 msgr2=0x7f970c13c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:20.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 -- 172.21.15.49:0/3629753070 shutdown_connections 2024-05-15T13:10:20.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 -- 172.21.15.49:0/3629753070 wait complete. 2024-05-15T13:10:20.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.036+0000 7f97134fa640 1 librados: shutdown 2024-05-15T13:10:20.037 INFO:tasks.ceph:Scrubbing osd.1 2024-05-15T13:10:20.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:10:20.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.155+0000 7f99ed890640 10 monclient: get_monmap_and_config 2024-05-15T13:10:20.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.155+0000 7f99ed890640 10 monclient: build_initial_monmap 2024-05-15T13:10:20.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.155+0000 7f99ed890640 10 monclient: monmap: 2024-05-15T13:10:20.145 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:20.145 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:20.146 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:20.156369+0000 2024-05-15T13:10:20.146 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:20.156369+0000 2024-05-15T13:10:20.146 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:20.146 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.146 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:20.146 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:20.146 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:20.146 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:20.146 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:20.146 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.156+0000 7f99ed890640 1 Processor -- start 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.156+0000 7f99ed890640 1 -- start start 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.156+0000 7f99ed890640 10 monclient: init 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f99e806b860 0x7f99e8071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 10 monclient(hunting): picked mon.noname-b con 0x7f99e806b860 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f99e806c9c0 0x7f99e806bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 10 monclient(hunting): picked mon.noname-c con 0x7f99e806c9c0 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99e8142e30 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e8071350 con 0x7f99e8142e30 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e81429a0 con 0x7f99e806c9c0 2024-05-15T13:10:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e81401f0 con 0x7f99e806b860 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99ed890640 10 monclient(hunting): authenticate will time out at 3778.318833s 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99e77fe640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f99e8142e30 0x7f99e813fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:55248/0 (socket says 172.21.15.49:55248) 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.157+0000 7f99e77fe640 1 -- 172.21.15.49:0/1482184375 learned_addr learned my addr 172.21.15.49:0/1482184375 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e6ffd640 1 --2- 172.21.15.49:0/1482184375 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f99e806b860 0x7f99e8071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e67fc640 1 --2- 172.21.15.49:0/1482184375 >> v2:172.21.15.88:3300/0 conn(0x7f99e806c9c0 0x7f99e806bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e6ffd640 10 monclient(hunting): get_auth_request con 0x7f99e806b860 auth_method 0 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e67fc640 10 monclient(hunting): get_auth_request con 0x7f99e806c9c0 auth_method 0 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e67fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:20.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:20.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:20.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.158+0000 7f99e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:20.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.159+0000 7f99e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:20.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.159+0000 7f99e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:20.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.159+0000 7f99e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:20.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.159+0000 7f99e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e5ffb640 1 -- 172.21.15.49:0/1482184375 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672491291 0 0) 0x7f99e8071350 con 0x7f99e8142e30 2024-05-15T13:10:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e67fc640 10 monclient(hunting): handle_auth_done global_id 22015 payload 274 2024-05-15T13:10:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e67fc640 1 -- 172.21.15.49:0/1482184375 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f99e806b860 msgr2=0x7f99e8071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e67fc640 1 --2- 172.21.15.49:0/1482184375 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f99e806b860 0x7f99e8071700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e67fc640 1 -- 172.21.15.49:0/1482184375 >> v1:172.21.15.49:6789/0 conn(0x7f99e8142e30 legacy=0x7f99e813fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e67fc640 10 monclient: _finish_hunting 0 2024-05-15T13:10:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e67fc640 1 monclient: found mon.noname-c 2024-05-15T13:10:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T13:10:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e67fc640 1 -- 172.21.15.49:0/1482184375 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e8140370 con 0x7f99e806c9c0 2024-05-15T13:10:20.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.160+0000 7f99e5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-05-15T13:10:20.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.162+0000 7f99e67fc640 1 --2- 172.21.15.49:0/1482184375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e806c9c0 0x7f99e806bc40 secure :-1 s=READY pgs=5450 cs=0 l=1 rev1=1 crypto rx=0x7f99d00097a0 tx=0x7f99d002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=281b6a291f40574a server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.164+0000 7f99e5ffb640 1 -- 172.21.15.49:0/1482184375 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0038070 con 0x7f99e806c9c0 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.164+0000 7f99e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.164+0000 7f99e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.164+0000 7f99e5ffb640 10 monclient: dump: 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:20.155 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:20.155 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:20.155 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.164+0000 7f99e5ffb640 10 monclient: _finish_auth 0 2024-05-15T13:10:20.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.164+0000 7f99e5ffb640 10 monclient: _check_auth_tickets 2024-05-15T13:10:20.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.164+0000 7f99e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:20.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.164+0000 7f99ed890640 5 monclient: authenticate success, global_id 22015 2024-05-15T13:10:20.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.164+0000 7f99ed890640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:20.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99e5ffb640 1 -- 172.21.15.49:0/1482184375 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d0031700 con 0x7f99e806c9c0 2024-05-15T13:10:20.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:20.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99e5ffb640 1 -- 172.21.15.49:0/1482184375 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0031ad0 con 0x7f99e806c9c0 2024-05-15T13:10:20.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:20.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 4 set_mon_vals no callback set 2024-05-15T13:10:20.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99e5ffb640 10 monclient: dump: 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 10 monclient: shutdown 2024-05-15T13:10:20.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:20.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 1 -- 172.21.15.49:0/1482184375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e806c9c0 msgr2=0x7f99e806bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 1 --2- 172.21.15.49:0/1482184375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e806c9c0 0x7f99e806bc40 secure :-1 s=READY pgs=5450 cs=0 l=1 rev1=1 crypto rx=0x7f99d00097a0 tx=0x7f99d002b7a0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 1 -- 172.21.15.49:0/1482184375 shutdown_connections 2024-05-15T13:10:20.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 1 --2- 172.21.15.49:0/1482184375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e806c9c0 0x7f99e806bc40 unknown :-1 s=CLOSED pgs=5450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 1 --2- 172.21.15.49:0/1482184375 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f99e806b860 0x7f99e8071700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 1 -- 172.21.15.49:0/1482184375 >> 172.21.15.49:0/1482184375 conn(0x7f99e80b6370 msgr2=0x7f99e80b6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:20.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 1 -- 172.21.15.49:0/1482184375 shutdown_connections 2024-05-15T13:10:20.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.166+0000 7f99ed890640 1 -- 172.21.15.49:0/1482184375 wait complete. 2024-05-15T13:10:20.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.167+0000 7f99ed890640 10 monclient: build_initial_monmap 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.167+0000 7f99ed890640 10 monclient: monmap: 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.167+0000 7f99ed890640 1 librados: starting msgr at 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.167+0000 7f99ed890640 1 librados: starting objecter 2024-05-15T13:10:20.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 1 Processor -- start 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 1 -- start start 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 1 librados: setting wanted keys 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 1 librados: calling monclient init 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient: init 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e806b860 0x7f99e81421f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient(hunting): picked mon.noname-b con 0x7f99e806b860 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99e806c9c0 0x7f99e8140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient(hunting): picked mon.noname-a con 0x7f99e806c9c0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f99e8142e30 0x7f99e8140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient(hunting): picked mon.noname-c con 0x7f99e8142e30 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e8143690 con 0x7f99e806c9c0 2024-05-15T13:10:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e8143510 con 0x7f99e806b860 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e814d080 con 0x7f99e8142e30 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.168+0000 7f99ed890640 10 monclient(hunting): authenticate will time out at 3778.330317s 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e77fe640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f99e8142e30 0x7f99e8140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e67fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99e806c9c0 0x7f99e8140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e806b860 0x7f99e81421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e67fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99e806c9c0 0x7f99e8140850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59924/0 (socket says 172.21.15.49:59924) 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e67fc640 1 -- 172.21.15.49:0/1959923315 learned_addr learned my addr 172.21.15.49:0/1959923315 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e67fc640 10 monclient(hunting): get_auth_request con 0x7f99e806c9c0 auth_method 0 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e67fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:20.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e77fe640 10 monclient(hunting): get_auth_request con 0x7f99e8142e30 auth_method 0 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e77fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e6ffd640 10 monclient(hunting): get_auth_request con 0x7f99e806b860 auth_method 0 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.169+0000 7f99e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.170+0000 7f99e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.170+0000 7f99e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.170+0000 7f99e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.170+0000 7f99e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.170+0000 7f99e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.170+0000 7f99e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:20.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.170+0000 7f99e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:20.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.170+0000 7f99e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:20.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.170+0000 7f99e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.172+0000 7f99e6ffd640 10 monclient(hunting): handle_auth_done global_id 22018 payload 824 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.172+0000 7f99e6ffd640 1 -- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f99e8142e30 msgr2=0x7f99e8140d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.172+0000 7f99e6ffd640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f99e8142e30 0x7f99e8140d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.172+0000 7f99e6ffd640 1 -- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99e806c9c0 msgr2=0x7f99e8140850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.172+0000 7f99e6ffd640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99e806c9c0 0x7f99e8140850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.172+0000 7f99e6ffd640 10 monclient: _finish_hunting 0 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.172+0000 7f99e6ffd640 1 monclient: found mon.noname-b 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.172+0000 7f99e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.172+0000 7f99e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.172+0000 7f99e6ffd640 1 -- 172.21.15.49:0/1959923315 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e813f530 con 0x7f99e806b860 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.174+0000 7f99e6ffd640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e806b860 0x7f99e81421f0 secure :-1 s=READY pgs=5451 cs=0 l=1 rev1=1 crypto rx=0x7f99d800e850 tx=0x7f99d800ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.174+0000 7f99dffff640 10 client.?.objecter ms_handle_connect 0x7f99e806b860 2024-05-15T13:10:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.174+0000 7f99dffff640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:20.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 1 -- 172.21.15.49:0/1959923315 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8017070 con 0x7f99e806b860 2024-05-15T13:10:20.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:20.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:20.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 10 monclient: dump: 2024-05-15T13:10:20.169 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:20.169 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:20.169 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.169 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.169 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 10 monclient: _finish_auth 0 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 10 monclient: _check_auth_tickets 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99ed890640 5 monclient: authenticate success, global_id 22018 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 1 -- 172.21.15.49:0/1959923315 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d800c940 con 0x7f99e806b860 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99ed890640 10 monclient: _renew_subs 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99ed890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e80b0cb0 con 0x7f99e806b860 2024-05-15T13:10:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 1 -- 172.21.15.49:0/1959923315 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d800cd10 con 0x7f99e806b860 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99dffff640 10 monclient: dump: 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:20.171 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99e57fa640 4 set_mon_vals no callback set 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99ed890640 10 client.22018.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99ed890640 10 monclient: _renew_subs 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99ed890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e81414b0 con 0x7f99e806b860 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.178+0000 7f99ed890640 1 librados: init done 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.179+0000 7f99ed890640 10 client.22018.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.179+0000 7f99ed890640 15 client.22018.objecter _session_command_op_assign -1 1 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.179+0000 7f99ed890640 20 client.22018.objecter _get_session osd=-1 returning homeless 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.179+0000 7f99ed890640 10 client.22018.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.179+0000 7f99ed890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:20.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.179+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f99e806cf30 con 0x7f99e806b860 2024-05-15T13:10:20.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.201+0000 7f99dffff640 1 -- 172.21.15.49:0/1959923315 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f99d801a070 con 0x7f99e806b860 2024-05-15T13:10:20.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.201+0000 7f99dffff640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f99c806ceb0 0x7f99c806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.201+0000 7f99e67fc640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f99c806ceb0 0x7f99c806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.201+0000 7f99e67fc640 10 monclient: get_auth_request con 0x7f99c806ceb0 auth_method 0 2024-05-15T13:10:20.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.202+0000 7f99dffff640 10 client.22018.objecter ms_handle_connect 0x7f99c806ceb0 2024-05-15T13:10:20.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.202+0000 7f99e67fc640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f99c806ceb0 0x7f99c806f370 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f99d0000c00 tx=0x7f99d0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:20.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 1 -- 172.21.15.49:0/1959923315 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f99d8056d30 con 0x7f99e806b860 2024-05-15T13:10:20.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 10 client.22018.objecter ms_dispatch 0x7f99e8000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 3 client.22018.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 3 client.22018.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 10 client.22018.objecter checking command 1 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] conn(0x7f99c8075900 0x7f99c8077da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 20 client.22018.objecter _get_session s=0x7f99c8075230 osd=1 3 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 20 client.22018.objecter put_session s=0x7f99c8075230 osd=1 3 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 15 client.22018.objecter _session_command_op_remove -1 1 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 20 client.22018.objecter _get_session s=0x7f99c8075230 osd=1 3 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 20 client.22018.objecter get_session s=0x7f99c8075230 osd=1 3 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 15 client.22018.objecter _session_command_op_assign 1 1 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 20 client.22018.objecter put_session s=0x7f99c8075230 osd=1 4 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 10 client.22018.objecter _send_command 1 2024-05-15T13:10:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 1 -- 172.21.15.49:0/1959923315 --> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99c8078470 con 0x7f99c8075900 2024-05-15T13:10:20.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.203+0000 7f99dffff640 20 client.22018.objecter dump_active .. 0 homeless 2024-05-15T13:10:20.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.204+0000 7f99dffff640 1 -- 172.21.15.49:0/1959923315 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=1251) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99d8054f30 con 0x7f99e806b860 2024-05-15T13:10:20.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.204+0000 7f99dffff640 10 monclient: handle_get_version_reply finishing 1 version 1251 2024-05-15T13:10:20.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.204+0000 7f99e77fe640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] conn(0x7f99c8075900 0x7f99c8077da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.204+0000 7f99e77fe640 10 monclient: get_auth_request con 0x7f99c8075900 auth_method 0 2024-05-15T13:10:20.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.207+0000 7f99e77fe640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] conn(0x7f99c8075900 0x7f99c8077da0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:20.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.207+0000 7f99dffff640 10 client.22018.objecter ms_handle_connect 0x7f99c8075900 2024-05-15T13:10:20.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.232+0000 7f99dffff640 1 -- 172.21.15.49:0/1959923315 <== osd.1 v2:172.21.15.49:6810/3250791401 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f99c8078470 con 0x7f99c8075900 2024-05-15T13:10:20.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.232+0000 7f99dffff640 10 client.22018.objecter ms_dispatch 0x7f99e8000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:10:20.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.232+0000 7f99dffff640 10 client.22018.objecter _finish_command 1 = system:0 2024-05-15T13:10:20.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.232+0000 7f99dffff640 20 client.22018.objecter put_session s=0x7f99c8075230 osd=1 4 2024-05-15T13:10:20.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.232+0000 7f99dffff640 15 client.22018.objecter _session_command_op_remove 1 1 2024-05-15T13:10:20.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 10 client.22018.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:10:20.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 15 client.22018.objecter _session_command_op_assign -1 2 2024-05-15T13:10:20.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 20 client.22018.objecter _get_session s=0x7f99c8075230 osd=1 3 2024-05-15T13:10:20.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 20 client.22018.objecter put_session s=0x7f99c8075230 osd=1 3 2024-05-15T13:10:20.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 20 client.22018.objecter _get_session s=0x7f99c8075230 osd=1 3 2024-05-15T13:10:20.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 15 client.22018.objecter _session_command_op_remove -1 2 2024-05-15T13:10:20.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 20 client.22018.objecter get_session s=0x7f99c8075230 osd=1 3 2024-05-15T13:10:20.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 15 client.22018.objecter _session_command_op_assign 1 2 2024-05-15T13:10:20.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 20 client.22018.objecter put_session s=0x7f99c8075230 osd=1 4 2024-05-15T13:10:20.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 10 client.22018.objecter _send_command 2 2024-05-15T13:10:20.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.246+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 --> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f99e81438a0 con 0x7f99c8075900 2024-05-15T13:10:20.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99dffff640 1 -- 172.21.15.49:0/1959923315 <== osd.1 v2:172.21.15.49:6810/3250791401 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f99e81438a0 con 0x7f99c8075900 2024-05-15T13:10:20.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99dffff640 10 client.22018.objecter ms_dispatch 0x7f99e8000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:10:20.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99dffff640 10 client.22018.objecter _finish_command 2 = system:0 2024-05-15T13:10:20.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99dffff640 20 client.22018.objecter put_session s=0x7f99c8075230 osd=1 4 2024-05-15T13:10:20.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99dffff640 15 client.22018.objecter _session_command_op_remove 1 2 2024-05-15T13:10:20.249 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-05-15T13:10:20.249 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:10:20.249 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-05-15T13:10:20.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 10 librados: watch_flush enter 2024-05-15T13:10:20.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 10 librados: watch_flush exit 2024-05-15T13:10:20.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 10 client.22018.objecter close_session for osd.1 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] conn(0x7f99c8075900 msgr2=0x7f99c8077da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:6810/3250791401,v1:172.21.15.49:6811/3250791401] conn(0x7f99c8075900 0x7f99c8077da0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 20 client.22018.objecter put_session s=0x7f99c8075230 osd=1 1 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 20 client.22018.objecter shutdown clearing up homeless session... 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 10 client.22018.objecter successfully canceled tick 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f99c806ceb0 msgr2=0x7f99c806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f99c806ceb0 0x7f99c806f370 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f99d0000c00 tx=0x7f99d0008030 comp rx=0 tx=0).stop 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 10 monclient: shutdown 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e806b860 msgr2=0x7f99e81421f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 1 --2- 172.21.15.49:0/1959923315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99e806b860 0x7f99e81421f0 secure :-1 s=READY pgs=5451 cs=0 l=1 rev1=1 crypto rx=0x7f99d800e850 tx=0x7f99d800ed10 comp rx=0 tx=0).stop 2024-05-15T13:10:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99e77fe640 1 -- 172.21.15.49:0/1959923315 reap_dead start 2024-05-15T13:10:20.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 shutdown_connections 2024-05-15T13:10:20.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.259+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 >> 172.21.15.49:0/1959923315 conn(0x7f99e80b6370 msgr2=0x7f99e8137b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:20.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.260+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 shutdown_connections 2024-05-15T13:10:20.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.260+0000 7f99ed890640 1 -- 172.21.15.49:0/1959923315 wait complete. 2024-05-15T13:10:20.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.260+0000 7f99ed890640 1 librados: shutdown 2024-05-15T13:10:20.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-15T13:10:20.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.376+0000 7fc034b28640 10 monclient: get_monmap_and_config 2024-05-15T13:10:20.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.376+0000 7fc034b28640 10 monclient: build_initial_monmap 2024-05-15T13:10:20.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.376+0000 7fc034b28640 10 monclient: monmap: 2024-05-15T13:10:20.366 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:20.366 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:20.367 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:20.377366+0000 2024-05-15T13:10:20.367 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:20.377366+0000 2024-05-15T13:10:20.367 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:20.367 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.367 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:20.367 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:20.367 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:20.367 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:20.367 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:20.367 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.377+0000 7fc034b28640 1 Processor -- start 2024-05-15T13:10:20.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.377+0000 7fc034b28640 1 -- start start 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.377+0000 7fc034b28640 10 monclient: init 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0300b1290 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0300716d0 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc030148f20 0x7fc03014b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 10 monclient(hunting): picked mon.noname-c con 0x7fc030148f20 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc030072b00 con 0x7fc0300716d0 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc030072de0 con 0x7fc0300b1290 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0300b1670 con 0x7fc030148f20 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc034b28640 10 monclient(hunting): authenticate will time out at 3778.539798s 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc02dd9b640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fc0300716d0 0x7fc030144090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:55262/0 (socket says 172.21.15.49:55262) 2024-05-15T13:10:20.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.378+0000 7fc02dd9b640 1 -- 172.21.15.49:0/1839825179 learned_addr learned my addr 172.21.15.49:0/1839825179 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:20.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.379+0000 7fc02ed9d640 1 --2- 172.21.15.49:0/1839825179 >> v2:172.21.15.88:3300/0 conn(0x7fc030148f20 0x7fc03014b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.379+0000 7fc02ed9d640 10 monclient(hunting): get_auth_request con 0x7fc030148f20 auth_method 0 2024-05-15T13:10:20.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.379+0000 7fc02ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:20.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.379+0000 7fc02ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:20.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.379+0000 7fc02ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:20.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.380+0000 7fc02ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:20.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.380+0000 7fc02ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:20.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.380+0000 7fc02ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:20.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.381+0000 7fc02d59a640 1 -- 172.21.15.49:0/1839825179 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609416549 0 0) 0x7fc030072b00 con 0x7fc0300716d0 2024-05-15T13:10:20.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.381+0000 7fc02d59a640 10 monclient(hunting): _negotiate 2024-05-15T13:10:20.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.381+0000 7fc02d59a640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:20.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.381+0000 7fc02d59a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:20.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.381+0000 7fc02d59a640 10 monclient(hunting): my global_id is 16182 2024-05-15T13:10:20.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.381+0000 7fc02d59a640 1 -- 172.21.15.49:0/1839825179 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc018003520 con 0x7fc0300716d0 2024-05-15T13:10:20.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.382+0000 7fc02ed9d640 10 monclient(hunting): handle_auth_done global_id 22021 payload 274 2024-05-15T13:10:20.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.382+0000 7fc02ed9d640 1 -- 172.21.15.49:0/1839825179 >> v1:172.21.15.88:6789/0 conn(0x7fc0300b1290 legacy=0x7fc030143980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.382+0000 7fc02ed9d640 1 -- 172.21.15.49:0/1839825179 >> v1:172.21.15.49:6789/0 conn(0x7fc0300716d0 legacy=0x7fc030144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.382+0000 7fc02ed9d640 10 monclient: _finish_hunting 0 2024-05-15T13:10:20.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.382+0000 7fc02ed9d640 1 monclient: found mon.noname-c 2024-05-15T13:10:20.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.382+0000 7fc02ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:20.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.382+0000 7fc02ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T13:10:20.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.382+0000 7fc02ed9d640 1 -- 172.21.15.49:0/1839825179 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc030071c60 con 0x7fc030148f20 2024-05-15T13:10:20.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.384+0000 7fc02ed9d640 1 --2- 172.21.15.49:0/1839825179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc030148f20 0x7fc03014b310 secure :-1 s=READY pgs=5453 cs=0 l=1 rev1=1 crypto rx=0x7fc028009d90 tx=0x7fc02802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6c7b6c4e8445c36 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:20.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 1 -- 172.21.15.49:0/1839825179 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028039070 con 0x7fc030148f20 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 10 monclient: dump: 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:20.379 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 10 monclient: _finish_auth 0 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 10 monclient: _check_auth_tickets 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc034b28640 5 monclient: authenticate success, global_id 22021 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 1 -- 172.21.15.49:0/1839825179 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02802cc60 con 0x7fc030148f20 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc034b28640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 1 -- 172.21.15.49:0/1839825179 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028034660 con 0x7fc030148f20 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc034b28640 4 set_mon_vals no callback set 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc02d59a640 10 monclient: dump: 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:20.380 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.381 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.381 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:20.381 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.381 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:20.381 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:20.381 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:20.381 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc034b28640 10 monclient: shutdown 2024-05-15T13:10:20.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc034b28640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:20.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc034b28640 1 -- 172.21.15.49:0/1839825179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc030148f20 msgr2=0x7fc03014b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.389+0000 7fc034b28640 1 --2- 172.21.15.49:0/1839825179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc030148f20 0x7fc03014b310 secure :-1 s=READY pgs=5453 cs=0 l=1 rev1=1 crypto rx=0x7fc028009d90 tx=0x7fc02802beb0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.390+0000 7fc034b28640 1 -- 172.21.15.49:0/1839825179 shutdown_connections 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.390+0000 7fc034b28640 1 --2- 172.21.15.49:0/1839825179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc030148f20 0x7fc03014b310 unknown :-1 s=CLOSED pgs=5453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.390+0000 7fc034b28640 1 -- 172.21.15.49:0/1839825179 >> 172.21.15.49:0/1839825179 conn(0x7fc0300b4440 msgr2=0x7fc0300b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.390+0000 7fc034b28640 1 -- 172.21.15.49:0/1839825179 shutdown_connections 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.390+0000 7fc034b28640 1 -- 172.21.15.49:0/1839825179 wait complete. 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient: build_initial_monmap 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient: monmap: 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.385 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 librados: starting msgr at 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 librados: starting objecter 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 Processor -- start 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 -- start start 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 librados: setting wanted keys 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 librados: calling monclient init 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient: init 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300716d0 0x7fc030137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient(hunting): picked mon.noname-b con 0x7fc0300716d0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0300b1290 0x7fc03013e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient(hunting): picked mon.noname-a con 0x7fc0300b1290 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc030148f20 0x7fc03013e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient(hunting): picked mon.noname-c con 0x7fc030148f20 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03014bbe0 con 0x7fc0300b1290 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03014ba60 con 0x7fc0300716d0 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03014bd60 con 0x7fc030148f20 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.391+0000 7fc034b28640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc034b28640 10 monclient(hunting): authenticate will time out at 3778.553441s 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300716d0 0x7fc030137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0300b1290 0x7fc03013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02ed9d640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc030148f20 0x7fc03013e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300716d0 0x7fc030137020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43140/0 (socket says 172.21.15.49:43140) 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02e59c640 1 -- 172.21.15.49:0/4067363485 learned_addr learned my addr 172.21.15.49:0/4067363485 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02e59c640 10 monclient(hunting): get_auth_request con 0x7fc0300716d0 auth_method 0 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02e59c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02e59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02dd9b640 10 monclient(hunting): get_auth_request con 0x7fc0300b1290 auth_method 0 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02dd9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.392+0000 7fc02dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02ed9d640 10 monclient(hunting): get_auth_request con 0x7fc030148f20 auth_method 0 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:20.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:20.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:20.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:20.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:20.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:20.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:20.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:20.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:20.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.393+0000 7fc02ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:20.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02dd9b640 10 monclient(hunting): handle_auth_done global_id 16185 payload 824 2024-05-15T13:10:20.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02dd9b640 1 -- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc030148f20 msgr2=0x7fc03013e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02dd9b640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc030148f20 0x7fc03013e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02dd9b640 1 -- 172.21.15.49:0/4067363485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300716d0 msgr2=0x7fc030137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02dd9b640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300716d0 0x7fc030137020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02dd9b640 10 monclient: _finish_hunting 0 2024-05-15T13:10:20.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02dd9b640 1 monclient: found mon.noname-a 2024-05-15T13:10:20.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:20.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T13:10:20.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02dd9b640 1 -- 172.21.15.49:0/4067363485 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc030138a70 con 0x7fc0300b1290 2024-05-15T13:10:20.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.396+0000 7fc02e59c640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300716d0 0x7fc030137020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-15T13:10:20.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.398+0000 7fc02dd9b640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0300b1290 0x7fc03013e280 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7fc02400c430 tx=0x7fc02400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:20.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.398+0000 7fc01effd640 10 client.?.objecter ms_handle_connect 0x7fc0300b1290 2024-05-15T13:10:20.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.398+0000 7fc01effd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 1 -- 172.21.15.49:0/4067363485 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024017070 con 0x7fc0300b1290 2024-05-15T13:10:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:10:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 10 monclient: dump: 2024-05-15T13:10:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 10 monclient: _finish_auth 0 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 10 monclient: _check_auth_tickets 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 1 -- 172.21.15.49:0/4067363485 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc024005ce0 con 0x7fc0300b1290 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 1 -- 172.21.15.49:0/4067363485 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0240072d0 con 0x7fc0300b1290 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc034b28640 5 monclient: authenticate success, global_id 16185 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc034b28640 10 monclient: _renew_subs 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc034b28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc034b28640 1 -- 172.21.15.49:0/4067363485 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc030138d50 con 0x7fc0300b1290 2024-05-15T13:10:20.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc01effd640 10 monclient: dump: 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc02cd99640 4 set_mon_vals no callback set 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc034b28640 10 client.16185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc034b28640 10 monclient: _renew_subs 2024-05-15T13:10:20.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc034b28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T13:10:20.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc034b28640 1 -- 172.21.15.49:0/4067363485 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03013ed00 con 0x7fc0300b1290 2024-05-15T13:10:20.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.404+0000 7fc034b28640 1 librados: init done 2024-05-15T13:10:20.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.405+0000 7fc034b28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:20.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.405+0000 7fc034b28640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:20.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.405+0000 7fc034b28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T13:10:20.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.405+0000 7fc034b28640 1 -- 172.21.15.49:0/4067363485 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03004f0e0 con 0x7fc0300b1290 2024-05-15T13:10:20.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.674+0000 7fc01effd640 1 -- 172.21.15.49:0/4067363485 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc02401a070 con 0x7fc0300b1290 2024-05-15T13:10:20.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.675+0000 7fc01effd640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc01006cee0 0x7fc01006f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:20.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.675+0000 7fc02e59c640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc01006cee0 0x7fc01006f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:20.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.675+0000 7fc02e59c640 10 monclient: get_auth_request con 0x7fc01006cee0 auth_method 0 2024-05-15T13:10:20.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.675+0000 7fc02e59c640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc01006cee0 0x7fc01006f3a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc020007fd0 tx=0x7fc020009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:20.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.675+0000 7fc01effd640 10 client.16185.objecter ms_handle_connect 0x7fc01006cee0 2024-05-15T13:10:20.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.719+0000 7fc01effd640 1 -- 172.21.15.49:0/4067363485 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fc0240562f0 con 0x7fc0300b1290 2024-05-15T13:10:20.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.719+0000 7fc01effd640 10 client.16185.objecter ms_dispatch 0x7fc030000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:20.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.719+0000 7fc01effd640 3 client.16185.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:20.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.719+0000 7fc01effd640 3 client.16185.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:20.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.719+0000 7fc01effd640 20 client.16185.objecter dump_active .. 0 homeless 2024-05-15T13:10:20.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.719+0000 7fc01effd640 1 -- 172.21.15.49:0/4067363485 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc024058250 con 0x7fc0300b1290 2024-05-15T13:10:20.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.719+0000 7fc01effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:20.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.719+0000 7fc01effd640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:20.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.907+0000 7fc034b28640 1 -- 172.21.15.49:0/4067363485 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc03006b0c0 con 0x7fc01006cee0 2024-05-15T13:10:20.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.907+0000 7fc01effd640 1 -- 172.21.15.49:0/4067363485 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc03006b0c0 con 0x7fc01006cee0 2024-05-15T13:10:20.897 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 1 to deep-scrub 2024-05-15T13:10:20.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 10 librados: watch_flush enter 2024-05-15T13:10:20.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 10 librados: watch_flush exit 2024-05-15T13:10:20.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 20 client.16185.objecter shutdown clearing up homeless session... 2024-05-15T13:10:20.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 10 client.16185.objecter successfully canceled tick 2024-05-15T13:10:20.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 -- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc01006cee0 msgr2=0x7fc01006f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc01006cee0 0x7fc01006f3a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc020007fd0 tx=0x7fc020009040 comp rx=0 tx=0).stop 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 10 monclient: shutdown 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 -- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0300b1290 msgr2=0x7fc03013e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0300b1290 0x7fc03013e280 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7fc02400c430 tx=0x7fc02400c8f0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 -- 172.21.15.49:0/4067363485 shutdown_connections 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc030148f20 0x7fc03013e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0300b1290 0x7fc03013e280 unknown :-1 s=CLOSED pgs=3263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc01006cee0 0x7fc01006f3a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 --2- 172.21.15.49:0/4067363485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300716d0 0x7fc030137020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 -- 172.21.15.49:0/4067363485 >> 172.21.15.49:0/4067363485 conn(0x7fc0300b4440 msgr2=0x7fc0300aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:20.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 -- 172.21.15.49:0/4067363485 shutdown_connections 2024-05-15T13:10:20.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 -- 172.21.15.49:0/4067363485 wait complete. 2024-05-15T13:10:20.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:20.909+0000 7fc034b28640 1 librados: shutdown 2024-05-15T13:10:20.911 INFO:tasks.ceph:Scrubbing osd.2 2024-05-15T13:10:20.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.030+0000 7f364413a640 10 monclient: get_monmap_and_config 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.030+0000 7f364413a640 10 monclient: build_initial_monmap 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.030+0000 7f364413a640 10 monclient: monmap: 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:21.030612+0000 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:21.030612+0000 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:21.020 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:21.021 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:21.021 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:21.021 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:21.021 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.030+0000 7f364413a640 1 Processor -- start 2024-05-15T13:10:21.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.030+0000 7f364413a640 1 -- start start 2024-05-15T13:10:21.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.030+0000 7f364413a640 10 monclient: init 2024-05-15T13:10:21.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f363c0b5410 0x7f363c145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 10 monclient(hunting): picked mon.noname-c con 0x7f363c0b5410 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f363c0aa0f0 0x7f363c13f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 10 monclient(hunting): picked mon.noname-b con 0x7f363c0aa0f0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f363c13fcd0 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c0ab220 con 0x7f363c13fcd0 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f363c0b57f0 con 0x7f363c0b5410 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0a9810 con 0x7f363c0aa0f0 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:21.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.031+0000 7f364413a640 10 monclient(hunting): authenticate will time out at 3779.193026s 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f36416d5640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f363c0aa0f0 0x7f363c13f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f3641ed6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f363c0b5410 0x7f363c145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f36426d7640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f363c13fcd0 0x7f363c0b2bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:55218/0 (socket says 172.21.15.49:55218) 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f36426d7640 1 -- 172.21.15.49:0/2719806464 learned_addr learned my addr 172.21.15.49:0/2719806464 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f36416d5640 10 monclient(hunting): get_auth_request con 0x7f363c0aa0f0 auth_method 0 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f36416d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f36416d5640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f36416d5640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f3641ed6640 10 monclient(hunting): get_auth_request con 0x7f363c0b5410 auth_method 0 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f3641ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f3641ed6640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.032+0000 7f3641ed6640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.033+0000 7f36416d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.033+0000 7f36416d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.033+0000 7f36416d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.033+0000 7f3641ed6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.033+0000 7f3641ed6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.033+0000 7f3641ed6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.034+0000 7f36416d5640 10 monclient(hunting): handle_auth_done global_id 17621 payload 274 2024-05-15T13:10:21.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.034+0000 7f36416d5640 1 -- 172.21.15.49:0/2719806464 >> v2:172.21.15.88:3300/0 conn(0x7f363c0b5410 msgr2=0x7f363c145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:21.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.034+0000 7f36416d5640 1 --2- 172.21.15.49:0/2719806464 >> v2:172.21.15.88:3300/0 conn(0x7f363c0b5410 0x7f363c145bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.034+0000 7f36416d5640 1 -- 172.21.15.49:0/2719806464 >> v1:172.21.15.88:6789/0 conn(0x7f363c13fcd0 legacy=0x7f363c0b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.034+0000 7f36416d5640 10 monclient: _finish_hunting 0 2024-05-15T13:10:21.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.034+0000 7f36416d5640 1 monclient: found mon.noname-b 2024-05-15T13:10:21.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.034+0000 7f36416d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:21.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.034+0000 7f36416d5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T13:10:21.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.034+0000 7f36416d5640 1 -- 172.21.15.49:0/2719806464 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c146110 con 0x7f363c0aa0f0 2024-05-15T13:10:21.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.036+0000 7f36416d5640 1 --2- 172.21.15.49:0/2719806464 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f363c0aa0f0 0x7f363c13f790 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7f36300097a0 tx=0x7f363002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8512ab0905216e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 1 -- 172.21.15.49:0/2719806464 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36300315a0 con 0x7f363c0aa0f0 2024-05-15T13:10:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 10 monclient: dump: 2024-05-15T13:10:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 10 monclient: _finish_auth 0 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 10 monclient: _check_auth_tickets 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 1 -- 172.21.15.49:0/2719806464 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3630037070 con 0x7f363c0aa0f0 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 1 -- 172.21.15.49:0/2719806464 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630031960 con 0x7f363c0aa0f0 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f3640ed4640 10 monclient: dump: 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.032 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f364413a640 5 monclient: authenticate success, global_id 17621 2024-05-15T13:10:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f364413a640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.040+0000 7f364413a640 4 set_mon_vals no callback set 2024-05-15T13:10:21.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 10 monclient: shutdown 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 1 -- 172.21.15.49:0/2719806464 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f363c0aa0f0 msgr2=0x7f363c13f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 1 --2- 172.21.15.49:0/2719806464 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f363c0aa0f0 0x7f363c13f790 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7f36300097a0 tx=0x7f363002b840 comp rx=0 tx=0).stop 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 1 -- 172.21.15.49:0/2719806464 shutdown_connections 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 1 --2- 172.21.15.49:0/2719806464 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f363c0aa0f0 0x7f363c13f790 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 1 --2- 172.21.15.49:0/2719806464 >> v2:172.21.15.88:3300/0 conn(0x7f363c0b5410 0x7f363c145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 1 -- 172.21.15.49:0/2719806464 >> 172.21.15.49:0/2719806464 conn(0x7f363c066f00 msgr2=0x7f363c067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 1 -- 172.21.15.49:0/2719806464 shutdown_connections 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 1 -- 172.21.15.49:0/2719806464 wait complete. 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 10 monclient: build_initial_monmap 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.041+0000 7f364413a640 10 monclient: monmap: 2024-05-15T13:10:21.037 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.042+0000 7f364413a640 1 librados: starting msgr at 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.042+0000 7f364413a640 1 librados: starting objecter 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.042+0000 7f364413a640 1 Processor -- start 2024-05-15T13:10:21.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.042+0000 7f364413a640 1 -- start start 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.042+0000 7f364413a640 1 librados: setting wanted keys 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.042+0000 7f364413a640 1 librados: calling monclient init 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.042+0000 7f364413a640 10 monclient: init 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f363c0aa0f0 0x7f363c0ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 10 monclient(hunting): picked mon.noname-a con 0x7f363c0aa0f0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f363c0b5410 0x7f363c0ac7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 10 monclient(hunting): picked mon.noname-c con 0x7f363c0b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363c13fcd0 0x7f363c139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 10 monclient(hunting): picked mon.noname-b con 0x7f363c13fcd0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0a9bd0 con 0x7f363c0aa0f0 2024-05-15T13:10:21.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0ab770 con 0x7f363c13fcd0 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0aa530 con 0x7f363c0b5410 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f364413a640 10 monclient(hunting): authenticate will time out at 3779.204516s 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f36416d5640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f363c0b5410 0x7f363c0ac7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f3641ed6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f363c0aa0f0 0x7f363c0ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.043+0000 7f36426d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363c13fcd0 0x7f363c139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f3641ed6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f363c0aa0f0 0x7f363c0ad080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59938/0 (socket says 172.21.15.49:59938) 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36426d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363c13fcd0 0x7f363c139470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43158/0 (socket says 172.21.15.49:43158) 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f3641ed6640 1 -- 172.21.15.49:0/439749268 learned_addr learned my addr 172.21.15.49:0/439749268 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f3641ed6640 10 monclient(hunting): get_auth_request con 0x7f363c0aa0f0 auth_method 0 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f3641ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f3641ed6640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f3641ed6640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36426d7640 10 monclient(hunting): get_auth_request con 0x7f363c13fcd0 auth_method 0 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36426d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36426d7640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36426d7640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36416d5640 10 monclient(hunting): get_auth_request con 0x7f363c0b5410 auth_method 0 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36416d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36416d5640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36416d5640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36426d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36426d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36426d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f3641ed6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f3641ed6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f3641ed6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36416d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36416d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.044+0000 7f36416d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.047+0000 7f36426d7640 10 monclient(hunting): handle_auth_done global_id 22033 payload 824 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.047+0000 7f36426d7640 1 -- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f363c0b5410 msgr2=0x7f363c0ac7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.047+0000 7f36426d7640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f363c0b5410 0x7f363c0ac7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.047+0000 7f36426d7640 1 -- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f363c0aa0f0 msgr2=0x7f363c0ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.047+0000 7f36426d7640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f363c0aa0f0 0x7f363c0ad080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.047+0000 7f36426d7640 10 monclient: _finish_hunting 0 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.047+0000 7f36426d7640 1 monclient: found mon.noname-b 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.047+0000 7f36426d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.047+0000 7f36426d7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.047+0000 7f36426d7640 1 -- 172.21.15.49:0/439749268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c0b1a30 con 0x7f363c13fcd0 2024-05-15T13:10:21.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.049+0000 7f36426d7640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363c13fcd0 0x7f363c139470 secure :-1 s=READY pgs=5455 cs=0 l=1 rev1=1 crypto rx=0x7f363400c3f0 tx=0x7f363400c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.049+0000 7f363a7fc640 10 client.?.objecter ms_handle_connect 0x7f363c13fcd0 2024-05-15T13:10:21.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.049+0000 7f363a7fc640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:21.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 1 -- 172.21.15.49:0/439749268 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634017070 con 0x7f363c13fcd0 2024-05-15T13:10:21.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:21.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 10 monclient: dump: 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 10 monclient: _finish_auth 0 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 10 monclient: _check_auth_tickets 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 1 -- 172.21.15.49:0/439749268 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3634005ce0 con 0x7f363c13fcd0 2024-05-15T13:10:21.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f364413a640 5 monclient: authenticate success, global_id 22033 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f364413a640 10 monclient: _renew_subs 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f364413a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363c0acd00 con 0x7f363c13fcd0 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 1 -- 172.21.15.49:0/439749268 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36340072f0 con 0x7f363c13fcd0 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363a7fc640 10 monclient: dump: 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.046 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f363bfff640 4 set_mon_vals no callback set 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f364413a640 10 client.22033.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f364413a640 10 monclient: _renew_subs 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f364413a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f363c139b90 con 0x7f363c13fcd0 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.052+0000 7f364413a640 1 librados: init done 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.053+0000 7f364413a640 10 client.22033.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:21.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.053+0000 7f364413a640 15 client.22033.objecter _session_command_op_assign -1 1 2024-05-15T13:10:21.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.053+0000 7f364413a640 20 client.22033.objecter _get_session osd=-1 returning homeless 2024-05-15T13:10:21.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.053+0000 7f364413a640 10 client.22033.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:21.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.053+0000 7f364413a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.053+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f363c04f0e0 con 0x7f363c13fcd0 2024-05-15T13:10:21.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.072+0000 7f363a7fc640 1 -- 172.21.15.49:0/439749268 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f363401a070 con 0x7f363c13fcd0 2024-05-15T13:10:21.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.073+0000 7f363a7fc640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f362806ce60 0x7f362806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.073+0000 7f3641ed6640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f362806ce60 0x7f362806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.073+0000 7f3641ed6640 10 monclient: get_auth_request con 0x7f362806ce60 auth_method 0 2024-05-15T13:10:21.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.073+0000 7f363a7fc640 10 client.22033.objecter ms_handle_connect 0x7f362806ce60 2024-05-15T13:10:21.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.073+0000 7f3641ed6640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f362806ce60 0x7f362806f320 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f362c009a40 tx=0x7f362c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 1 -- 172.21.15.49:0/439749268 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f3634056370 con 0x7f363c13fcd0 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 10 client.22033.objecter ms_dispatch 0x7f363c000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 3 client.22033.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 3 client.22033.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 10 client.22033.objecter checking command 1 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x7f3628075850 0x7f3628077cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 20 client.22033.objecter _get_session s=0x7f3628075180 osd=2 3 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 20 client.22033.objecter put_session s=0x7f3628075180 osd=2 3 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 15 client.22033.objecter _session_command_op_remove -1 1 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 20 client.22033.objecter _get_session s=0x7f3628075180 osd=2 3 2024-05-15T13:10:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 20 client.22033.objecter get_session s=0x7f3628075180 osd=2 3 2024-05-15T13:10:21.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 15 client.22033.objecter _session_command_op_assign 2 1 2024-05-15T13:10:21.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 20 client.22033.objecter put_session s=0x7f3628075180 osd=2 4 2024-05-15T13:10:21.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 10 client.22033.objecter _send_command 1 2024-05-15T13:10:21.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 1 -- 172.21.15.49:0/439749268 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36280783c0 con 0x7f3628075850 2024-05-15T13:10:21.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 20 client.22033.objecter dump_active .. 0 homeless 2024-05-15T13:10:21.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 1 -- 172.21.15.49:0/439749268 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=1251) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3634054570 con 0x7f363c13fcd0 2024-05-15T13:10:21.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.075+0000 7f363a7fc640 10 monclient: handle_get_version_reply finishing 1 version 1251 2024-05-15T13:10:21.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.076+0000 7f36416d5640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x7f3628075850 0x7f3628077cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.076+0000 7f36416d5640 10 monclient: get_auth_request con 0x7f3628075850 auth_method 0 2024-05-15T13:10:21.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.079+0000 7f363a7fc640 10 client.22033.objecter ms_handle_connect 0x7f3628075850 2024-05-15T13:10:21.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.079+0000 7f36416d5640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x7f3628075850 0x7f3628077cf0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.105+0000 7f363a7fc640 1 -- 172.21.15.49:0/439749268 <== osd.2 v2:172.21.15.49:6826/2361777220 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f36280783c0 con 0x7f3628075850 2024-05-15T13:10:21.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.105+0000 7f363a7fc640 10 client.22033.objecter ms_dispatch 0x7f363c000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:10:21.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.105+0000 7f363a7fc640 10 client.22033.objecter _finish_command 1 = system:0 2024-05-15T13:10:21.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.105+0000 7f363a7fc640 20 client.22033.objecter put_session s=0x7f3628075180 osd=2 4 2024-05-15T13:10:21.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.105+0000 7f363a7fc640 15 client.22033.objecter _session_command_op_remove 2 1 2024-05-15T13:10:21.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 10 client.22033.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:10:21.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 15 client.22033.objecter _session_command_op_assign -1 2 2024-05-15T13:10:21.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 20 client.22033.objecter _get_session s=0x7f3628075180 osd=2 3 2024-05-15T13:10:21.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 20 client.22033.objecter put_session s=0x7f3628075180 osd=2 3 2024-05-15T13:10:21.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 20 client.22033.objecter _get_session s=0x7f3628075180 osd=2 3 2024-05-15T13:10:21.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 15 client.22033.objecter _session_command_op_remove -1 2 2024-05-15T13:10:21.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 20 client.22033.objecter get_session s=0x7f3628075180 osd=2 3 2024-05-15T13:10:21.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 15 client.22033.objecter _session_command_op_assign 2 2 2024-05-15T13:10:21.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 20 client.22033.objecter put_session s=0x7f3628075180 osd=2 4 2024-05-15T13:10:21.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 10 client.22033.objecter _send_command 2 2024-05-15T13:10:21.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.119+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 --> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f363c139e80 con 0x7f3628075850 2024-05-15T13:10:21.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.133+0000 7f363a7fc640 1 -- 172.21.15.49:0/439749268 <== osd.2 v2:172.21.15.49:6826/2361777220 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f363c139e80 con 0x7f3628075850 2024-05-15T13:10:21.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.133+0000 7f363a7fc640 10 client.22033.objecter ms_dispatch 0x7f363c000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:10:21.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.133+0000 7f363a7fc640 10 client.22033.objecter _finish_command 2 = system:0 2024-05-15T13:10:21.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.133+0000 7f363a7fc640 20 client.22033.objecter put_session s=0x7f3628075180 osd=2 4 2024-05-15T13:10:21.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.133+0000 7f363a7fc640 15 client.22033.objecter _session_command_op_remove 2 2 2024-05-15T13:10:21.123 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-05-15T13:10:21.124 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:10:21.124 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-05-15T13:10:21.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 10 librados: watch_flush enter 2024-05-15T13:10:21.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 10 librados: watch_flush exit 2024-05-15T13:10:21.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 10 client.22033.objecter close_session for osd.2 2024-05-15T13:10:21.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x7f3628075850 msgr2=0x7f3628077cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:6826/2361777220,v1:172.21.15.49:6827/2361777220] conn(0x7f3628075850 0x7f3628077cf0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 20 client.22033.objecter put_session s=0x7f3628075180 osd=2 1 2024-05-15T13:10:21.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 20 client.22033.objecter shutdown clearing up homeless session... 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 10 client.22033.objecter successfully canceled tick 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f362806ce60 msgr2=0x7f362806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f362806ce60 0x7f362806f320 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f362c009a40 tx=0x7f362c00b040 comp rx=0 tx=0).stop 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 10 monclient: shutdown 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363c13fcd0 msgr2=0x7f363c139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 --2- 172.21.15.49:0/439749268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363c13fcd0 0x7f363c139470 secure :-1 s=READY pgs=5455 cs=0 l=1 rev1=1 crypto rx=0x7f363400c3f0 tx=0x7f363400c8b0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f36426d7640 1 -- 172.21.15.49:0/439749268 reap_dead start 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 shutdown_connections 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 >> 172.21.15.49:0/439749268 conn(0x7f363c066f00 msgr2=0x7f363c0672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 shutdown_connections 2024-05-15T13:10:21.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 -- 172.21.15.49:0/439749268 wait complete. 2024-05-15T13:10:21.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.134+0000 7f364413a640 1 librados: shutdown 2024-05-15T13:10:21.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-15T13:10:21.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.250+0000 7fd8bb835640 10 monclient: get_monmap_and_config 2024-05-15T13:10:21.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.250+0000 7fd8bb835640 10 monclient: build_initial_monmap 2024-05-15T13:10:21.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.251+0000 7fd8bb835640 10 monclient: monmap: 2024-05-15T13:10:21.240 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:21.240 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.240 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:21.251511+0000 2024-05-15T13:10:21.240 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:21.251511+0000 2024-05-15T13:10:21.241 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:21.241 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.241 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:21.241 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:21.241 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:21.241 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:21.241 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:21.241 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.251+0000 7fd8bb835640 1 Processor -- start 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.251+0000 7fd8bb835640 1 -- start start 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.251+0000 7fd8bb835640 10 monclient: init 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b40b5410 0x7fd8b40a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 10 monclient(hunting): picked mon.noname-b con 0x7fd8b40b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8b413f990 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd8b4145c00 0x7fd8b4146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 10 monclient(hunting): picked mon.noname-c con 0x7fd8b4145c00 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b40ab470 con 0x7fd8b413f990 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b40b57f0 con 0x7fd8b4145c00 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40a9ea0 con 0x7fd8b40b5410 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.252+0000 7fd8bb835640 10 monclient(hunting): authenticate will time out at 3779.414081s 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b8dd0640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fd8b413f990 0x7fd8b413fdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:55270/0 (socket says 172.21.15.49:55270) 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b8dd0640 1 -- 172.21.15.49:0/609817234 learned_addr learned my addr 172.21.15.49:0/609817234 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b9dd2640 1 --2- 172.21.15.49:0/609817234 >> v2:172.21.15.88:3300/0 conn(0x7fd8b4145c00 0x7fd8b4146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b95d1640 1 --2- 172.21.15.49:0/609817234 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b40b5410 0x7fd8b40a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b9dd2640 10 monclient(hunting): get_auth_request con 0x7fd8b4145c00 auth_method 0 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b9dd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b9dd2640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b9dd2640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b95d1640 10 monclient(hunting): get_auth_request con 0x7fd8b40b5410 auth_method 0 2024-05-15T13:10:21.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b95d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b95d1640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.253+0000 7fd8b95d1640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.254+0000 7fd8b9dd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.254+0000 7fd8b9dd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.254+0000 7fd8b9dd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.254+0000 7fd8b95d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.254+0000 7fd8b95d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.254+0000 7fd8b95d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.255+0000 7fd8b9dd2640 10 monclient(hunting): handle_auth_done global_id 22036 payload 274 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.255+0000 7fd8b9dd2640 1 -- 172.21.15.49:0/609817234 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b40b5410 msgr2=0x7fd8b40a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.255+0000 7fd8b9dd2640 1 --2- 172.21.15.49:0/609817234 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b40b5410 0x7fd8b40a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.255+0000 7fd8b9dd2640 1 -- 172.21.15.49:0/609817234 >> v1:172.21.15.49:6789/0 conn(0x7fd8b413f990 legacy=0x7fd8b413fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.255+0000 7fd8b9dd2640 10 monclient: _finish_hunting 0 2024-05-15T13:10:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.255+0000 7fd8b9dd2640 1 monclient: found mon.noname-c 2024-05-15T13:10:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.255+0000 7fd8b9dd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.255+0000 7fd8b9dd2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.255+0000 7fd8b9dd2640 1 -- 172.21.15.49:0/609817234 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b40aa0c0 con 0x7fd8b4145c00 2024-05-15T13:10:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.257+0000 7fd8b9dd2640 1 --2- 172.21.15.49:0/609817234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 0x7fd8b4146070 secure :-1 s=READY pgs=5456 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009dc0 tx=0x7fd8ac02be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b5c7637193aab64 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 1 -- 172.21.15.49:0/609817234 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac039070 con 0x7fd8b4145c00 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 10 monclient: dump: 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.251 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 10 monclient: _finish_auth 0 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 10 monclient: _check_auth_tickets 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 1 -- 172.21.15.49:0/609817234 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac02cbb0 con 0x7fd8b4145c00 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8bb835640 5 monclient: authenticate success, global_id 22036 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8bb835640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8bb835640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 1 -- 172.21.15.49:0/609817234 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac034440 con 0x7fd8b4145c00 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8bb835640 4 set_mon_vals no callback set 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.261+0000 7fd8a7fff640 10 monclient: dump: 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.253 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.253 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.253 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.253 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.253 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.253 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.253 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.253 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 10 monclient: shutdown 2024-05-15T13:10:21.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:21.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 1 -- 172.21.15.49:0/609817234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 msgr2=0x7fd8b4146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 1 --2- 172.21.15.49:0/609817234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 0x7fd8b4146070 secure :-1 s=READY pgs=5456 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009dc0 tx=0x7fd8ac02be00 comp rx=0 tx=0).stop 2024-05-15T13:10:21.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 1 -- 172.21.15.49:0/609817234 shutdown_connections 2024-05-15T13:10:21.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 1 --2- 172.21.15.49:0/609817234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 0x7fd8b4146070 unknown :-1 s=CLOSED pgs=5456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 1 --2- 172.21.15.49:0/609817234 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b40b5410 0x7fd8b40a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 1 -- 172.21.15.49:0/609817234 >> 172.21.15.49:0/609817234 conn(0x7fd8b4066f00 msgr2=0x7fd8b4067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 1 -- 172.21.15.49:0/609817234 shutdown_connections 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 1 -- 172.21.15.49:0/609817234 wait complete. 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 10 monclient: build_initial_monmap 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.262+0000 7fd8bb835640 10 monclient: monmap: 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:21.257 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.263+0000 7fd8bb835640 1 librados: starting msgr at 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.263+0000 7fd8bb835640 1 librados: starting objecter 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.263+0000 7fd8bb835640 1 Processor -- start 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.263+0000 7fd8bb835640 1 -- start start 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.263+0000 7fd8bb835640 1 librados: setting wanted keys 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.263+0000 7fd8bb835640 1 librados: calling monclient init 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.263+0000 7fd8bb835640 10 monclient: init 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.263+0000 7fd8bb835640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.263+0000 7fd8bb835640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.263+0000 7fd8bb835640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b40b5410 0x7fd8b4142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 10 monclient(hunting): picked mon.noname-a con 0x7fd8b40b5410 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:21.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b413f990 0x7fd8b41421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 10 monclient(hunting): picked mon.noname-c con 0x7fd8b413f990 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 0x7fd8b4134a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 10 monclient(hunting): picked mon.noname-b con 0x7fd8b4145c00 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40b4aa0 con 0x7fd8b40b5410 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40b4920 con 0x7fd8b4145c00 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b414cf30 con 0x7fd8b413f990 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8bb835640 10 monclient(hunting): authenticate will time out at 3779.425522s 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b95d1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b40b5410 0x7fd8b4142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b9dd2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 0x7fd8b4134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b8dd0640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b413f990 0x7fd8b41421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b9dd2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 0x7fd8b4134a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43188/0 (socket says 172.21.15.49:43188) 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b9dd2640 1 -- 172.21.15.49:0/2716556996 learned_addr learned my addr 172.21.15.49:0/2716556996 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b9dd2640 10 monclient(hunting): get_auth_request con 0x7fd8b4145c00 auth_method 0 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b9dd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b9dd2640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b9dd2640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b95d1640 10 monclient(hunting): get_auth_request con 0x7fd8b40b5410 auth_method 0 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b95d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b95d1640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.264+0000 7fd8b95d1640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b8dd0640 10 monclient(hunting): get_auth_request con 0x7fd8b413f990 auth_method 0 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b8dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b8dd0640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b8dd0640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b9dd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b9dd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b9dd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b95d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b95d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b95d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b8dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b8dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.265+0000 7fd8b8dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.268+0000 7fd8b9dd2640 10 monclient(hunting): handle_auth_done global_id 22039 payload 824 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.268+0000 7fd8b9dd2640 1 -- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b413f990 msgr2=0x7fd8b41421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.268+0000 7fd8b9dd2640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b413f990 0x7fd8b41421b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.268+0000 7fd8b9dd2640 1 -- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b40b5410 msgr2=0x7fd8b4142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.268+0000 7fd8b9dd2640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b40b5410 0x7fd8b4142730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.268+0000 7fd8b9dd2640 10 monclient: _finish_hunting 0 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.268+0000 7fd8b9dd2640 1 monclient: found mon.noname-b 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.268+0000 7fd8b9dd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.268+0000 7fd8b9dd2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.268+0000 7fd8b9dd2640 1 -- 172.21.15.49:0/2716556996 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b4143240 con 0x7fd8b4145c00 2024-05-15T13:10:21.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.270+0000 7fd8b9dd2640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 0x7fd8b4134a40 secure :-1 s=READY pgs=5457 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac037040 tx=0x7fd8ac004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.270+0000 7fd8a5ffb640 10 client.?.objecter ms_handle_connect 0x7fd8b4145c00 2024-05-15T13:10:21.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.270+0000 7fd8a5ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.273+0000 7fd8a5ffb640 1 -- 172.21.15.49:0/2716556996 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac039050 con 0x7fd8b4145c00 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.273+0000 7fd8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.273+0000 7fd8a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.273+0000 7fd8a5ffb640 10 monclient: dump: 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.273+0000 7fd8a5ffb640 10 monclient: _finish_auth 0 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.273+0000 7fd8a5ffb640 10 monclient: _check_auth_tickets 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.273+0000 7fd8a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8bb835640 5 monclient: authenticate success, global_id 22039 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8a5ffb640 1 -- 172.21.15.49:0/2716556996 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac045070 con 0x7fd8b4145c00 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8a5ffb640 1 -- 172.21.15.49:0/2716556996 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac009450 con 0x7fd8b4145c00 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8bb835640 10 monclient: _renew_subs 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8bb835640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8bb835640 1 -- 172.21.15.49:0/2716556996 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b4136ab0 con 0x7fd8b4145c00 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8a5ffb640 10 monclient: dump: 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8a77fe640 4 set_mon_vals no callback set 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8bb835640 10 client.22039.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:21.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8bb835640 10 monclient: _renew_subs 2024-05-15T13:10:21.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8bb835640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8bb835640 1 -- 172.21.15.49:0/2716556996 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b4135160 con 0x7fd8b4145c00 2024-05-15T13:10:21.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.274+0000 7fd8bb835640 1 librados: init done 2024-05-15T13:10:21.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.275+0000 7fd8bb835640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:21.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.275+0000 7fd8bb835640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:21.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.275+0000 7fd8bb835640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.275+0000 7fd8bb835640 1 -- 172.21.15.49:0/2716556996 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b404f0e0 con 0x7fd8b4145c00 2024-05-15T13:10:21.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.293+0000 7fd8a5ffb640 1 -- 172.21.15.49:0/2716556996 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd8ac03e850 con 0x7fd8b4145c00 2024-05-15T13:10:21.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.294+0000 7fd8a5ffb640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd89806ce60 0x7fd89806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.294+0000 7fd8b95d1640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd89806ce60 0x7fd89806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.294+0000 7fd8b95d1640 10 monclient: get_auth_request con 0x7fd89806ce60 auth_method 0 2024-05-15T13:10:21.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.294+0000 7fd8b95d1640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd89806ce60 0x7fd89806f320 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009b40 tx=0x7fd8b0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.294+0000 7fd8a5ffb640 10 client.22039.objecter ms_handle_connect 0x7fd89806ce60 2024-05-15T13:10:21.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.296+0000 7fd8a5ffb640 1 -- 172.21.15.49:0/2716556996 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fd8ac07a180 con 0x7fd8b4145c00 2024-05-15T13:10:21.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.296+0000 7fd8a5ffb640 10 client.22039.objecter ms_dispatch 0x7fd8b4000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:21.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.296+0000 7fd8a5ffb640 3 client.22039.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:21.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.296+0000 7fd8a5ffb640 3 client.22039.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:21.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.296+0000 7fd8a5ffb640 20 client.22039.objecter dump_active .. 0 homeless 2024-05-15T13:10:21.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.532+0000 7fd8a5ffb640 1 -- 172.21.15.49:0/2716556996 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd8ac07c0e0 con 0x7fd8b4145c00 2024-05-15T13:10:21.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.532+0000 7fd8a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:21.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.532+0000 7fd8a5ffb640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:21.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.717+0000 7fd8bb835640 1 -- 172.21.15.49:0/2716556996 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8b413fdb0 con 0x7fd89806ce60 2024-05-15T13:10:21.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.717+0000 7fd8a5ffb640 1 -- 172.21.15.49:0/2716556996 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd8b413fdb0 con 0x7fd89806ce60 2024-05-15T13:10:21.707 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 2 to deep-scrub 2024-05-15T13:10:21.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 10 librados: watch_flush enter 2024-05-15T13:10:21.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 10 librados: watch_flush exit 2024-05-15T13:10:21.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 20 client.22039.objecter shutdown clearing up homeless session... 2024-05-15T13:10:21.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 10 client.22039.objecter successfully canceled tick 2024-05-15T13:10:21.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 -- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd89806ce60 msgr2=0x7fd89806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd89806ce60 0x7fd89806f320 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009b40 tx=0x7fd8b0009190 comp rx=0 tx=0).stop 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 10 monclient: shutdown 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 -- 172.21.15.49:0/2716556996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 msgr2=0x7fd8b4134a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 0x7fd8b4134a40 secure :-1 s=READY pgs=5457 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac037040 tx=0x7fd8ac004a40 comp rx=0 tx=0).stop 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 -- 172.21.15.49:0/2716556996 shutdown_connections 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fd89806ce60 0x7fd89806f320 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4145c00 0x7fd8b4134a40 unknown :-1 s=CLOSED pgs=5457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b413f990 0x7fd8b41421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 --2- 172.21.15.49:0/2716556996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b40b5410 0x7fd8b4142730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 -- 172.21.15.49:0/2716556996 >> 172.21.15.49:0/2716556996 conn(0x7fd8b4066f00 msgr2=0x7fd8b413a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 -- 172.21.15.49:0/2716556996 shutdown_connections 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 -- 172.21.15.49:0/2716556996 wait complete. 2024-05-15T13:10:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.719+0000 7fd8bb835640 1 librados: shutdown 2024-05-15T13:10:21.721 INFO:tasks.ceph:Scrubbing osd.3 2024-05-15T13:10:21.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:10:21.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.842+0000 7f5f602e3640 10 monclient: get_monmap_and_config 2024-05-15T13:10:21.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.842+0000 7f5f602e3640 10 monclient: build_initial_monmap 2024-05-15T13:10:21.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.842+0000 7f5f602e3640 10 monclient: monmap: 2024-05-15T13:10:21.832 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:21.832 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.832 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:21.842990+0000 2024-05-15T13:10:21.832 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:21.842990+0000 2024-05-15T13:10:21.833 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:21.833 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.833 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:21.833 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:21.833 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:21.833 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:21.833 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:21.833 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 1 Processor -- start 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 1 -- start start 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 10 monclient: init 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5f580b8a90 0x7f5f580aa410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 10 monclient(hunting): picked mon.noname-a con 0x7f5f580b8a90 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f580a99d0 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f580b1ef0 0x7f5f5813fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 10 monclient(hunting): picked mon.noname-b con 0x7f5f580b1ef0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:21.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f58146410 con 0x7f5f580a99d0 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f580b8e70 con 0x7f5f580b8a90 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580ab5f0 con 0x7f5f580b1ef0 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.843+0000 7f5f602e3640 10 monclient(hunting): authenticate will time out at 3780.005394s 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.844+0000 7f5f5e07f640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5f580b8a90 0x7f5f580aa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.844+0000 7f5f5e07f640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5f580b8a90 0x7f5f580aa410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59952/0 (socket says 172.21.15.49:59952) 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.844+0000 7f5f5e07f640 1 -- 172.21.15.49:0/1397594083 learned_addr learned my addr 172.21.15.49:0/1397594083 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.844+0000 7f5f5e07f640 10 monclient(hunting): get_auth_request con 0x7f5f580b8a90 auth_method 0 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.844+0000 7f5f5e07f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.844+0000 7f5f5e07f640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.844+0000 7f5f5e07f640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.845+0000 7f5f5e07f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.845+0000 7f5f5e07f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.845+0000 7f5f5e07f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.846+0000 7f5f5e07f640 10 monclient(hunting): handle_auth_done global_id 16197 payload 274 2024-05-15T13:10:21.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.846+0000 7f5f5e07f640 1 -- 172.21.15.49:0/1397594083 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f580b1ef0 msgr2=0x7f5f5813fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:21.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.846+0000 7f5f5e07f640 1 --2- 172.21.15.49:0/1397594083 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f580b1ef0 0x7f5f5813fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.847+0000 7f5f5e07f640 1 -- 172.21.15.49:0/1397594083 >> v1:172.21.15.88:6789/0 conn(0x7f5f580a99d0 legacy=0x7f5f58145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:21.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.847+0000 7f5f5e07f640 10 monclient: _finish_hunting 0 2024-05-15T13:10:21.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.847+0000 7f5f5e07f640 1 monclient: found mon.noname-a 2024-05-15T13:10:21.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.847+0000 7f5f5e07f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:21.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.847+0000 7f5f5e07f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T13:10:21.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.847+0000 7f5f5e07f640 1 -- 172.21.15.49:0/1397594083 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f580a9ef0 con 0x7f5f580b8a90 2024-05-15T13:10:21.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.849+0000 7f5f5e07f640 1 --2- 172.21.15.49:0/1397594083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f580b8a90 0x7f5f580aa410 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009870 tx=0x7f5f5402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ef10d5c2bb78b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 1 -- 172.21.15.49:0/1397594083 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54035020 con 0x7f5f580b8a90 2024-05-15T13:10:21.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:10:21.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 10 monclient: dump: 2024-05-15T13:10:21.846 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.846 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.846 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.846 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.846 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.846 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 10 monclient: _finish_auth 0 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 10 monclient: _check_auth_tickets 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 1 -- 172.21.15.49:0/1397594083 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f54031640 con 0x7f5f580b8a90 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 1 -- 172.21.15.49:0/1397594083 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5402d040 con 0x7f5f580b8a90 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f5d07d640 10 monclient: dump: 2024-05-15T13:10:21.847 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f602e3640 5 monclient: authenticate success, global_id 16197 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f602e3640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.856+0000 7f5f602e3640 4 set_mon_vals no callback set 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.857+0000 7f5f602e3640 10 monclient: shutdown 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.857+0000 7f5f602e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.857+0000 7f5f602e3640 1 -- 172.21.15.49:0/1397594083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f580b8a90 msgr2=0x7f5f580aa410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.857+0000 7f5f602e3640 1 --2- 172.21.15.49:0/1397594083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f580b8a90 0x7f5f580aa410 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009870 tx=0x7f5f5402b8f0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.857+0000 7f5f602e3640 1 -- 172.21.15.49:0/1397594083 shutdown_connections 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.857+0000 7f5f602e3640 1 --2- 172.21.15.49:0/1397594083 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f580b1ef0 0x7f5f5813fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.857+0000 7f5f602e3640 1 --2- 172.21.15.49:0/1397594083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f580b8a90 0x7f5f580aa410 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.857+0000 7f5f602e3640 1 -- 172.21.15.49:0/1397594083 >> 172.21.15.49:0/1397594083 conn(0x7f5f58067110 msgr2=0x7f5f580b8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.857+0000 7f5f602e3640 1 -- 172.21.15.49:0/1397594083 shutdown_connections 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.857+0000 7f5f602e3640 1 -- 172.21.15.49:0/1397594083 wait complete. 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.858+0000 7f5f602e3640 10 monclient: build_initial_monmap 2024-05-15T13:10:21.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.858+0000 7f5f602e3640 10 monclient: monmap: 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.858+0000 7f5f602e3640 1 librados: starting msgr at 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.858+0000 7f5f602e3640 1 librados: starting objecter 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.858+0000 7f5f602e3640 1 Processor -- start 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.858+0000 7f5f602e3640 1 -- start start 2024-05-15T13:10:21.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.858+0000 7f5f602e3640 1 librados: setting wanted keys 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.858+0000 7f5f602e3640 1 librados: calling monclient init 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.858+0000 7f5f602e3640 10 monclient: init 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f580a99d0 0x7f5f580b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 10 monclient(hunting): picked mon.noname-a con 0x7f5f580a99d0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f580b1ef0 0x7f5f580ad610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 10 monclient(hunting): picked mon.noname-b con 0x7f5f580b1ef0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f580b8a90 0x7f5f580ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 10 monclient(hunting): picked mon.noname-c con 0x7f5f580b8a90 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5814d090 con 0x7f5f580a99d0 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f58140380 con 0x7f5f580b1ef0 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5814d210 con 0x7f5f580b8a90 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f602e3640 10 monclient(hunting): authenticate will time out at 3780.020638s 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f5d87e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f580b1ef0 0x7f5f580ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f5d87e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f580b1ef0 0x7f5f580ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43190/0 (socket says 172.21.15.49:43190) 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f5d87e640 1 -- 172.21.15.49:0/249690178 learned_addr learned my addr 172.21.15.49:0/249690178 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f5d87e640 10 monclient(hunting): get_auth_request con 0x7f5f580b1ef0 auth_method 0 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f5d87e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f5d87e640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.859+0000 7f5f5d87e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.860+0000 7f5f5e880640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f580b8a90 0x7f5f580ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.860+0000 7f5f5d87e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.860+0000 7f5f5d87e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.860+0000 7f5f5d87e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.860+0000 7f5f5e880640 10 monclient(hunting): get_auth_request con 0x7f5f580b8a90 auth_method 0 2024-05-15T13:10:21.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.860+0000 7f5f5e880640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:21.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.860+0000 7f5f5e880640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:21.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.860+0000 7f5f5e880640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:21.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.861+0000 7f5f5e880640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:21.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.861+0000 7f5f5e880640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:21.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.861+0000 7f5f5e880640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:21.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.862+0000 7f5f5e07f640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f580a99d0 0x7f5f580b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.863+0000 7f5f5d87e640 10 monclient(hunting): handle_auth_done global_id 22042 payload 824 2024-05-15T13:10:21.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.863+0000 7f5f5d87e640 1 -- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f580b8a90 msgr2=0x7f5f580ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.863+0000 7f5f5d87e640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f580b8a90 0x7f5f580ac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.863+0000 7f5f5d87e640 1 -- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f580a99d0 msgr2=0x7f5f580b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.863+0000 7f5f5d87e640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f580a99d0 0x7f5f580b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.863+0000 7f5f5d87e640 10 monclient: _finish_hunting 0 2024-05-15T13:10:21.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.863+0000 7f5f5d87e640 1 monclient: found mon.noname-b 2024-05-15T13:10:21.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.863+0000 7f5f5d87e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:21.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.863+0000 7f5f5d87e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.863+0000 7f5f5d87e640 1 -- 172.21.15.49:0/249690178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f580ace80 con 0x7f5f580b1ef0 2024-05-15T13:10:21.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.865+0000 7f5f5d87e640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f580b1ef0 0x7f5f580ad610 secure :-1 s=READY pgs=5458 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800c400 tx=0x7f5f4800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.865+0000 7f5f4effd640 10 client.?.objecter ms_handle_connect 0x7f5f580b1ef0 2024-05-15T13:10:21.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.865+0000 7f5f4effd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 1 -- 172.21.15.49:0/249690178 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f48017070 con 0x7f5f580b1ef0 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 10 monclient: dump: 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.860 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 10 monclient: _finish_auth 0 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 10 monclient: _check_auth_tickets 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 5 monclient: authenticate success, global_id 22042 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 1 -- 172.21.15.49:0/249690178 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f480035f0 con 0x7f5f580b1ef0 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 1 -- 172.21.15.49:0/249690178 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f48003a20 con 0x7f5f580b1ef0 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 10 monclient: _renew_subs 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f580ae520 con 0x7f5f580b1ef0 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f4effd640 10 monclient: dump: 2024-05-15T13:10:21.861 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f5c87c640 4 set_mon_vals no callback set 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 10 client.22042.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 10 monclient: _renew_subs 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f581416d0 con 0x7f5f580b1ef0 2024-05-15T13:10:21.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 1 librados: init done 2024-05-15T13:10:21.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 10 client.22042.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:21.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 15 client.22042.objecter _session_command_op_assign -1 1 2024-05-15T13:10:21.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 20 client.22042.objecter _get_session osd=-1 returning homeless 2024-05-15T13:10:21.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 10 client.22042.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:21.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:21.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.869+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f5813f980 con 0x7f5f580b1ef0 2024-05-15T13:10:21.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.887+0000 7f5f4effd640 1 -- 172.21.15.49:0/249690178 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5f4801a070 con 0x7f5f580b1ef0 2024-05-15T13:10:21.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.888+0000 7f5f4effd640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f5f3c06ce60 0x7f5f3c06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.888+0000 7f5f5e07f640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f5f3c06ce60 0x7f5f3c06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.888+0000 7f5f5e07f640 10 monclient: get_auth_request con 0x7f5f3c06ce60 auth_method 0 2024-05-15T13:10:21.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.888+0000 7f5f5e07f640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f5f3c06ce60 0x7f5f3c06f320 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5f54036000 tx=0x7f5f5403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.888+0000 7f5f4effd640 10 client.22042.objecter ms_handle_connect 0x7f5f3c06ce60 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 1 -- 172.21.15.49:0/249690178 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f5f48056950 con 0x7f5f580b1ef0 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 10 client.22042.objecter ms_dispatch 0x7f5f58000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 3 client.22042.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 3 client.22042.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 10 client.22042.objecter checking command 1 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] conn(0x7f5f3c075850 0x7f5f3c077cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 20 client.22042.objecter _get_session s=0x7f5f3c075180 osd=3 3 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 20 client.22042.objecter put_session s=0x7f5f3c075180 osd=3 3 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 15 client.22042.objecter _session_command_op_remove -1 1 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 20 client.22042.objecter _get_session s=0x7f5f3c075180 osd=3 3 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 20 client.22042.objecter get_session s=0x7f5f3c075180 osd=3 3 2024-05-15T13:10:21.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 15 client.22042.objecter _session_command_op_assign 3 1 2024-05-15T13:10:21.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 20 client.22042.objecter put_session s=0x7f5f3c075180 osd=3 4 2024-05-15T13:10:21.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 10 client.22042.objecter _send_command 1 2024-05-15T13:10:21.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 1 -- 172.21.15.49:0/249690178 --> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f4801ae50 con 0x7f5f3c075850 2024-05-15T13:10:21.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.891+0000 7f5f4effd640 20 client.22042.objecter dump_active .. 0 homeless 2024-05-15T13:10:21.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.892+0000 7f5f5e880640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] conn(0x7f5f3c075850 0x7f5f3c077cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:21.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.892+0000 7f5f5e880640 10 monclient: get_auth_request con 0x7f5f3c075850 auth_method 0 2024-05-15T13:10:21.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.895+0000 7f5f4effd640 10 client.22042.objecter ms_handle_connect 0x7f5f3c075850 2024-05-15T13:10:21.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.895+0000 7f5f5e880640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] conn(0x7f5f3c075850 0x7f5f3c077cf0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:21.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.940+0000 7f5f4effd640 1 -- 172.21.15.49:0/249690178 <== osd.3 v2:172.21.15.49:6802/945312262 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5f4801ae50 con 0x7f5f3c075850 2024-05-15T13:10:21.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.940+0000 7f5f4effd640 10 client.22042.objecter ms_dispatch 0x7f5f58000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:10:21.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.940+0000 7f5f4effd640 10 client.22042.objecter _finish_command 1 = system:0 2024-05-15T13:10:21.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.940+0000 7f5f4effd640 20 client.22042.objecter put_session s=0x7f5f3c075180 osd=3 4 2024-05-15T13:10:21.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.940+0000 7f5f4effd640 15 client.22042.objecter _session_command_op_remove 3 1 2024-05-15T13:10:21.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.948+0000 7f5f4effd640 1 -- 172.21.15.49:0/249690178 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=1251) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f48054b50 con 0x7f5f580b1ef0 2024-05-15T13:10:21.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.948+0000 7f5f4effd640 10 monclient: handle_get_version_reply finishing 1 version 1251 2024-05-15T13:10:21.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 10 client.22042.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:10:21.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 15 client.22042.objecter _session_command_op_assign -1 2 2024-05-15T13:10:21.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 20 client.22042.objecter _get_session s=0x7f5f3c075180 osd=3 3 2024-05-15T13:10:21.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 20 client.22042.objecter put_session s=0x7f5f3c075180 osd=3 3 2024-05-15T13:10:21.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 20 client.22042.objecter _get_session s=0x7f5f3c075180 osd=3 3 2024-05-15T13:10:21.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 15 client.22042.objecter _session_command_op_remove -1 2 2024-05-15T13:10:21.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 20 client.22042.objecter get_session s=0x7f5f3c075180 osd=3 3 2024-05-15T13:10:21.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 15 client.22042.objecter _session_command_op_assign 3 2 2024-05-15T13:10:21.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 20 client.22042.objecter put_session s=0x7f5f3c075180 osd=3 4 2024-05-15T13:10:21.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 10 client.22042.objecter _send_command 2 2024-05-15T13:10:21.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.956+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 --> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5f5804f0e0 con 0x7f5f3c075850 2024-05-15T13:10:21.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.972+0000 7f5f4effd640 1 -- 172.21.15.49:0/249690178 <== osd.3 v2:172.21.15.49:6802/945312262 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5f5804f0e0 con 0x7f5f3c075850 2024-05-15T13:10:21.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.972+0000 7f5f4effd640 10 client.22042.objecter ms_dispatch 0x7f5f58000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:10:21.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.972+0000 7f5f4effd640 10 client.22042.objecter _finish_command 2 = system:0 2024-05-15T13:10:21.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.972+0000 7f5f4effd640 20 client.22042.objecter put_session s=0x7f5f3c075180 osd=3 4 2024-05-15T13:10:21.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.972+0000 7f5f4effd640 15 client.22042.objecter _session_command_op_remove 3 2 2024-05-15T13:10:21.963 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-05-15T13:10:21.963 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:10:21.963 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-05-15T13:10:21.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 10 librados: watch_flush enter 2024-05-15T13:10:21.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 10 librados: watch_flush exit 2024-05-15T13:10:21.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 10 client.22042.objecter close_session for osd.3 2024-05-15T13:10:21.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] conn(0x7f5f3c075850 msgr2=0x7f5f3c077cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:6802/945312262,v1:172.21.15.49:6803/945312262] conn(0x7f5f3c075850 0x7f5f3c077cf0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 20 client.22042.objecter put_session s=0x7f5f3c075180 osd=3 1 2024-05-15T13:10:21.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 20 client.22042.objecter shutdown clearing up homeless session... 2024-05-15T13:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 10 client.22042.objecter successfully canceled tick 2024-05-15T13:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f5f3c06ce60 msgr2=0x7f5f3c06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f5f3c06ce60 0x7f5f3c06f320 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5f54036000 tx=0x7f5f5403c040 comp rx=0 tx=0).stop 2024-05-15T13:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 10 monclient: shutdown 2024-05-15T13:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f580b1ef0 msgr2=0x7f5f580ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 --2- 172.21.15.49:0/249690178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f580b1ef0 0x7f5f580ad610 secure :-1 s=READY pgs=5458 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800c400 tx=0x7f5f4800c8c0 comp rx=0 tx=0).stop 2024-05-15T13:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f5e880640 1 -- 172.21.15.49:0/249690178 reap_dead start 2024-05-15T13:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 shutdown_connections 2024-05-15T13:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 >> 172.21.15.49:0/249690178 conn(0x7f5f58067110 msgr2=0x7f5f580b5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:21.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 shutdown_connections 2024-05-15T13:10:21.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 -- 172.21.15.49:0/249690178 wait complete. 2024-05-15T13:10:21.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:21.973+0000 7f5f602e3640 1 librados: shutdown 2024-05-15T13:10:21.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-15T13:10:22.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.088+0000 7f7be4bbd640 10 monclient: get_monmap_and_config 2024-05-15T13:10:22.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.088+0000 7f7be4bbd640 10 monclient: build_initial_monmap 2024-05-15T13:10:22.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.089+0000 7f7be4bbd640 10 monclient: monmap: 2024-05-15T13:10:22.079 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:22.079 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.079 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:22.089505+0000 2024-05-15T13:10:22.079 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:22.089505+0000 2024-05-15T13:10:22.079 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:22.079 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.079 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:22.079 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:22.080 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:22.080 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:22.080 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:22.080 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.089+0000 7f7be4bbd640 1 Processor -- start 2024-05-15T13:10:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.089+0000 7f7be4bbd640 1 -- start start 2024-05-15T13:10:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.089+0000 7f7be4bbd640 10 monclient: init 2024-05-15T13:10:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7be00b5410 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7be00a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 10 monclient(hunting): picked mon.noname-b con 0x7f7be01400a0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7be0145c00 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be00ab470 con 0x7f7be0145c00 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be00aa0c0 con 0x7f7be00b5410 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be00b57f0 con 0x7f7be01400a0 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.090+0000 7f7be4bbd640 10 monclient(hunting): authenticate will time out at 3780.252044s 2024-05-15T13:10:22.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.091+0000 7f7bded9d640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f7be0145c00 0x7f7be0146070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:55274/0 (socket says 172.21.15.49:55274) 2024-05-15T13:10:22.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.091+0000 7f7bded9d640 1 -- 172.21.15.49:0/572103070 learned_addr learned my addr 172.21.15.49:0/572103070 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:22.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.091+0000 7f7bddd9b640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7be00a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.091+0000 7f7bddd9b640 1 -- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 msgr2=0x7f7be00a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-15T13:10:22.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.091+0000 7f7bddd9b640 1 -- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 msgr2=0x7f7be00a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T13:10:22.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.091+0000 7f7bddd9b640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7be00a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-15T13:10:22.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.091+0000 7f7bddd9b640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7be00a9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-15T13:10:22.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289680413 0 0) 0x7f7be00aa0c0 con 0x7f7be00b5410 2024-05-15T13:10:22.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 10 monclient(hunting): _negotiate 2024-05-15T13:10:22.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:22.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 10 monclient(hunting): my global_id is 22045 2024-05-15T13:10:22.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc8003520 con 0x7f7be00b5410 2024-05-15T13:10:22.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636285846 0 0) 0x7f7be00ab470 con 0x7f7be0145c00 2024-05-15T13:10:22.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 10 monclient(hunting): _negotiate 2024-05-15T13:10:22.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:22.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 10 monclient(hunting): my global_id is 16200 2024-05-15T13:10:22.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.093+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be00aa0c0 con 0x7f7be0145c00 2024-05-15T13:10:22.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77680353 0 0) 0x7f7bc8003520 con 0x7f7be00b5410 2024-05-15T13:10:22.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 msgr2=0x7f7be00a9960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T13:10:22.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7be00a9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 >> v1:172.21.15.49:6789/0 conn(0x7f7be0145c00 legacy=0x7f7be0146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient: _finish_hunting 0 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 monclient: found mon.noname-c-legacy 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.88:6789/0 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be00aa5e0 con 0x7f7be00b5410 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient: _finish_auth 0 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient: _check_auth_tickets 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2686122088 0 0) 0x7f7bd0001670 con 0x7f7be00b5410 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient: dump: 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] but i'm connected to v1:172.21.15.88:6789/0, reconnecting 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 >> v1:172.21.15.88:6789/0 conn(0x7f7be00b5410 legacy=0x7f7be013f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-15T13:10:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bc8003bb0 0x7f7bc8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient(hunting): picked mon.a con 0x7f7bc8003bb0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7bc8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient(hunting): picked mon.c con 0x7f7be01400a0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7bc8004e60 0x7f7bc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient(hunting): picked mon.b con 0x7f7bc8004e60 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be00b57f0 con 0x7f7bc8003bb0 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc8005620 con 0x7f7bc8004e60 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc800bf30 con 0x7f7be01400a0 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bdd59a640 10 monclient(hunting): _finish_auth 0 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.097+0000 7f7bddd9b640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7bc8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.098+0000 7f7bddd9b640 10 monclient(hunting): get_auth_request con 0x7f7be01400a0 auth_method 0 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.098+0000 7f7bddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.098+0000 7f7bddd9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.098+0000 7f7bddd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.098+0000 7f7bddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.098+0000 7f7bddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.098+0000 7f7bddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bded9d640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7bc8004e60 0x7f7bc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bded9d640 10 monclient(hunting): get_auth_request con 0x7f7bc8004e60 auth_method 0 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bded9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bded9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 10 monclient(hunting): handle_auth_done global_id 22045 payload 293 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 1 -- 172.21.15.49:0/572103070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7bc8004e60 msgr2=0x7f7bc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7bc8004e60 0x7f7bc800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 1 -- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bc8003bb0 msgr2=0x7f7bc8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bc8003bb0 0x7f7bc8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 10 monclient: _finish_hunting 0 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 1 monclient: found mon.c 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 1 -- 172.21.15.49:0/572103070 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc8003520 con 0x7f7be01400a0 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 10 monclient: _finish_auth 0 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 10 monclient: _check_auth_tickets 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7bddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7be4bbd640 5 monclient: authenticate success, global_id 22045 2024-05-15T13:10:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.100+0000 7f7be4bbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:22.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.102+0000 7f7bddd9b640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7bc8004800 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4006be0 tx=0x7f7bc8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3d3f90d53ba59fe server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:22.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4034070 con 0x7f7be01400a0 2024-05-15T13:10:22.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7bdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:22.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7bdd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7bdd59a640 10 monclient: dump: 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd4030640 con 0x7f7be01400a0 2024-05-15T13:10:22.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7bdd59a640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:22.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7be4bbd640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:22.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7bdd59a640 1 -- 172.21.15.49:0/572103070 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd400a590 con 0x7f7be01400a0 2024-05-15T13:10:22.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7be4bbd640 4 set_mon_vals no callback set 2024-05-15T13:10:22.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7bdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:22.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7bdd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:22.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.106+0000 7f7bdd59a640 10 monclient: dump: 2024-05-15T13:10:22.098 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:22.098 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.098 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.098 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.099 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:22.099 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.099 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:22.099 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:22.099 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:22.099 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.107+0000 7f7be4bbd640 10 monclient: shutdown 2024-05-15T13:10:22.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.107+0000 7f7be4bbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:22.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.107+0000 7f7be4bbd640 1 -- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 msgr2=0x7f7bc8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.107+0000 7f7be4bbd640 1 --2- 172.21.15.49:0/572103070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7bc8004800 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4006be0 tx=0x7f7bc8005830 comp rx=0 tx=0).stop 2024-05-15T13:10:22.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.107+0000 7f7bded9d640 1 -- 172.21.15.49:0/572103070 reap_dead start 2024-05-15T13:10:22.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.107+0000 7f7be4bbd640 1 -- 172.21.15.49:0/572103070 shutdown_connections 2024-05-15T13:10:22.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.107+0000 7f7be4bbd640 1 -- 172.21.15.49:0/572103070 >> 172.21.15.49:0/572103070 conn(0x7f7be0066f00 msgr2=0x7f7be0067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:22.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.107+0000 7f7be4bbd640 1 -- 172.21.15.49:0/572103070 shutdown_connections 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.107+0000 7f7be4bbd640 1 -- 172.21.15.49:0/572103070 wait complete. 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.108+0000 7f7be4bbd640 10 monclient: build_initial_monmap 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.108+0000 7f7be4bbd640 10 monclient: monmap: 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.108+0000 7f7be4bbd640 1 librados: starting msgr at 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.108+0000 7f7be4bbd640 1 librados: starting objecter 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 1 Processor -- start 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 1 -- start start 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 1 librados: setting wanted keys 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 1 librados: calling monclient init 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient: init 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T13:10:22.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7be00b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient(hunting): picked mon.noname-c con 0x7f7be01400a0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7be00abc70 0x7f7be01db7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient(hunting): picked mon.noname-b con 0x7f7be00abc70 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7be0144830 0x7f7be0144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient(hunting): picked mon.noname-a con 0x7f7be0144830 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be00a9960 con 0x7f7be0144830 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be00a9ea0 con 0x7f7be00abc70 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be00a9ae0 con 0x7f7be01400a0 2024-05-15T13:10:22.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.109+0000 7f7be4bbd640 10 monclient(hunting): authenticate will time out at 3780.271100s 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bded9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7be0144830 0x7f7be0144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bddd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7be00abc70 0x7f7be01db7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bde59c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7be00b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bddd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7be00abc70 0x7f7be01db7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43202/0 (socket says 172.21.15.49:43202) 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bddd9b640 1 -- 172.21.15.49:0/2152821682 learned_addr learned my addr 172.21.15.49:0/2152821682 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bddd9b640 10 monclient(hunting): get_auth_request con 0x7f7be00abc70 auth_method 0 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bddd9b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bded9d640 10 monclient(hunting): get_auth_request con 0x7f7be0144830 auth_method 0 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bded9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bded9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bde59c640 10 monclient(hunting): get_auth_request con 0x7f7be01400a0 auth_method 0 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bde59c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bde59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.110+0000 7f7bddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.111+0000 7f7bded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.111+0000 7f7bded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.111+0000 7f7bded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.111+0000 7f7bde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.111+0000 7f7bde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.111+0000 7f7bde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.113+0000 7f7bddd9b640 10 monclient(hunting): handle_auth_done global_id 22048 payload 824 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.113+0000 7f7bddd9b640 1 -- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 msgr2=0x7f7be00b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.113+0000 7f7bddd9b640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7be00b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.113+0000 7f7bddd9b640 1 -- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7be0144830 msgr2=0x7f7be0144ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.113+0000 7f7bddd9b640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7be0144830 0x7f7be0144ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.113+0000 7f7bddd9b640 10 monclient: _finish_hunting 0 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.113+0000 7f7bddd9b640 1 monclient: found mon.noname-b 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.113+0000 7f7bddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:22.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.113+0000 7f7bddd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:22.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.113+0000 7f7bddd9b640 1 -- 172.21.15.49:0/2152821682 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be00acf70 con 0x7f7be00abc70 2024-05-15T13:10:22.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.115+0000 7f7bddd9b640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7be00abc70 0x7f7be01db7b0 secure :-1 s=READY pgs=5460 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4006be0 tx=0x7f7bd4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:22.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.115+0000 7f7bceffd640 10 client.?.objecter ms_handle_connect 0x7f7be00abc70 2024-05-15T13:10:22.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.115+0000 7f7bceffd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:22.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 1 -- 172.21.15.49:0/2152821682 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4034040 con 0x7f7be00abc70 2024-05-15T13:10:22.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:22.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:22.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 10 monclient: dump: 2024-05-15T13:10:22.114 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:22.114 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 10 monclient: _finish_auth 0 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 10 monclient: _check_auth_tickets 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:22.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 1 -- 172.21.15.49:0/2152821682 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd4030ba0 con 0x7f7be00abc70 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7be4bbd640 5 monclient: authenticate success, global_id 22048 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 1 -- 172.21.15.49:0/2152821682 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4004030 con 0x7f7be00abc70 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bceffd640 10 monclient: dump: 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7bcffff640 4 set_mon_vals no callback set 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7be4bbd640 10 monclient: _renew_subs 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7be4bbd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7be4bbd640 1 -- 172.21.15.49:0/2152821682 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be01dc440 con 0x7f7be00abc70 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7be4bbd640 10 client.22048.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:22.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7be4bbd640 10 monclient: _renew_subs 2024-05-15T13:10:22.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7be4bbd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:22.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7be4bbd640 1 -- 172.21.15.49:0/2152821682 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be0145220 con 0x7f7be00abc70 2024-05-15T13:10:22.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.119+0000 7f7be4bbd640 1 librados: init done 2024-05-15T13:10:22.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.120+0000 7f7be4bbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:22.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.120+0000 7f7be4bbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:22.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.120+0000 7f7be4bbd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:22.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.120+0000 7f7be4bbd640 1 -- 172.21.15.49:0/2152821682 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be004f0e0 con 0x7f7be00abc70 2024-05-15T13:10:22.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.367+0000 7f7bceffd640 1 -- 172.21.15.49:0/2152821682 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7bd404b060 con 0x7f7be00abc70 2024-05-15T13:10:22.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.368+0000 7f7bceffd640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7bc006ceb0 0x7f7bc006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.368+0000 7f7bde59c640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7bc006ceb0 0x7f7bc006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.368+0000 7f7bde59c640 10 monclient: get_auth_request con 0x7f7bc006ceb0 auth_method 0 2024-05-15T13:10:22.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.368+0000 7f7bceffd640 10 client.22048.objecter ms_handle_connect 0x7f7bc006ceb0 2024-05-15T13:10:22.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.368+0000 7f7bde59c640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7bc006ceb0 0x7f7bc006f370 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0001400 tx=0x7f7bd0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:22.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.374+0000 7f7bceffd640 1 -- 172.21.15.49:0/2152821682 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f7bd4078390 con 0x7f7be00abc70 2024-05-15T13:10:22.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.374+0000 7f7bceffd640 10 client.22048.objecter ms_dispatch 0x7f7be0000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:22.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.374+0000 7f7bceffd640 3 client.22048.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:22.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.374+0000 7f7bceffd640 3 client.22048.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:22.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.374+0000 7f7bceffd640 20 client.22048.objecter dump_active .. 0 homeless 2024-05-15T13:10:22.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.383+0000 7f7bceffd640 1 -- 172.21.15.49:0/2152821682 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7bd407a2f0 con 0x7f7be00abc70 2024-05-15T13:10:22.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.383+0000 7f7bceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:22.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.383+0000 7f7bceffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:22.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.568+0000 7f7be4bbd640 1 -- 172.21.15.49:0/2152821682 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7be013d520 con 0x7f7bc006ceb0 2024-05-15T13:10:22.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.569+0000 7f7bceffd640 1 -- 172.21.15.49:0/2152821682 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7be013d520 con 0x7f7bc006ceb0 2024-05-15T13:10:22.558 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 3 to deep-scrub 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.570+0000 7f7be4bbd640 10 librados: watch_flush enter 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.570+0000 7f7be4bbd640 10 librados: watch_flush exit 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.570+0000 7f7be4bbd640 20 client.22048.objecter shutdown clearing up homeless session... 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.570+0000 7f7be4bbd640 10 client.22048.objecter successfully canceled tick 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.570+0000 7f7be4bbd640 1 -- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7bc006ceb0 msgr2=0x7f7bc006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.570+0000 7f7be4bbd640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7bc006ceb0 0x7f7bc006f370 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0001400 tx=0x7f7bd0009040 comp rx=0 tx=0).stop 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.570+0000 7f7be4bbd640 10 monclient: shutdown 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.570+0000 7f7be4bbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.570+0000 7f7be4bbd640 1 -- 172.21.15.49:0/2152821682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7be00abc70 msgr2=0x7f7be01db7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.570+0000 7f7be4bbd640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7be00abc70 0x7f7be01db7b0 secure :-1 s=READY pgs=5460 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4006be0 tx=0x7f7bd4004730 comp rx=0 tx=0).stop 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.571+0000 7f7be4bbd640 1 -- 172.21.15.49:0/2152821682 shutdown_connections 2024-05-15T13:10:22.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.571+0000 7f7be4bbd640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7be0144830 0x7f7be0144ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.571+0000 7f7be4bbd640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f7bc006ceb0 0x7f7bc006f370 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.571+0000 7f7be4bbd640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7be00abc70 0x7f7be01db7b0 unknown :-1 s=CLOSED pgs=5460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.571+0000 7f7be4bbd640 1 --2- 172.21.15.49:0/2152821682 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7be01400a0 0x7f7be00b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.571+0000 7f7be4bbd640 1 -- 172.21.15.49:0/2152821682 >> 172.21.15.49:0/2152821682 conn(0x7f7be0066f00 msgr2=0x7f7be013da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:22.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.571+0000 7f7be4bbd640 1 -- 172.21.15.49:0/2152821682 shutdown_connections 2024-05-15T13:10:22.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.571+0000 7f7be4bbd640 1 -- 172.21.15.49:0/2152821682 wait complete. 2024-05-15T13:10:22.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.571+0000 7f7be4bbd640 1 librados: shutdown 2024-05-15T13:10:22.572 INFO:tasks.ceph:Scrubbing osd.4 2024-05-15T13:10:22.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:10:22.668 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:51.877 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:22.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.698+0000 7f2a9b77e640 10 monclient: get_monmap_and_config 2024-05-15T13:10:22.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.698+0000 7f2a9b77e640 10 monclient: build_initial_monmap 2024-05-15T13:10:22.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.698+0000 7f2a9b77e640 10 monclient: monmap: 2024-05-15T13:10:22.688 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:22.688 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.688 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:22.699079+0000 2024-05-15T13:10:22.688 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:22.699079+0000 2024-05-15T13:10:22.688 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:22.688 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.689 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:22.689 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:22.689 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:22.689 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:22.689 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:22.689 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.699+0000 7f2a9b77e640 1 Processor -- start 2024-05-15T13:10:22.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.699+0000 7f2a9b77e640 1 -- start start 2024-05-15T13:10:22.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.699+0000 7f2a9b77e640 10 monclient: init 2024-05-15T13:10:22.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:22.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a940b5410 0x7f2a94145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 10 monclient(hunting): picked mon.noname-b con 0x7f2a940b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a940aa0f0 0x7f2a9413f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 10 monclient(hunting): picked mon.noname-c con 0x7f2a940aa0f0 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a9413fcd0 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a940ab220 con 0x7f2a9413fcd0 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a940b57f0 con 0x7f2a940aa0f0 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940a9810 con 0x7f2a940b5410 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:22.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.700+0000 7f2a9b77e640 10 monclient(hunting): authenticate will time out at 3780.861643s 2024-05-15T13:10:22.691 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:51.898 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:22.700 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:38:51.909 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:22.717 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:51.884 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:22.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.728+0000 7f2a98d19640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a940aa0f0 0x7f2a9413f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.731+0000 7f2a98d19640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a940aa0f0 0x7f2a9413f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43204/0 (socket says 172.21.15.49:43204) 2024-05-15T13:10:22.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.731+0000 7f2a98d19640 1 -- 172.21.15.49:0/767518384 learned_addr learned my addr 172.21.15.49:0/767518384 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:22.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.731+0000 7f2a98d19640 10 monclient(hunting): get_auth_request con 0x7f2a940aa0f0 auth_method 0 2024-05-15T13:10:22.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.731+0000 7f2a98d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:22.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.731+0000 7f2a98d19640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.731+0000 7f2a98d19640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:22.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.732+0000 7f2a98d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:22.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.732+0000 7f2a98d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:22.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.732+0000 7f2a98d19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:22.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.735+0000 7f2a98d19640 10 monclient(hunting): handle_auth_done global_id 22051 payload 274 2024-05-15T13:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.735+0000 7f2a98d19640 1 -- 172.21.15.49:0/767518384 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a940b5410 msgr2=0x7f2a94145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.735+0000 7f2a98d19640 1 --2- 172.21.15.49:0/767518384 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a940b5410 0x7f2a94145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.735+0000 7f2a98d19640 1 -- 172.21.15.49:0/767518384 >> v1:172.21.15.88:6789/0 conn(0x7f2a9413fcd0 legacy=0x7f2a940b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.735+0000 7f2a98d19640 10 monclient: _finish_hunting 0 2024-05-15T13:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.735+0000 7f2a98d19640 1 monclient: found mon.noname-c 2024-05-15T13:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.735+0000 7f2a98d19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.735+0000 7f2a98d19640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T13:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.735+0000 7f2a98d19640 1 -- 172.21.15.49:0/767518384 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a94146110 con 0x7f2a940aa0f0 2024-05-15T13:10:22.725 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:51.892 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:22.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.741+0000 7f2a98d19640 1 --2- 172.21.15.49:0/767518384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a940aa0f0 0x7f2a9413f790 secure :-1 s=READY pgs=5462 cs=0 l=1 rev1=1 crypto rx=0x7f2a84009870 tx=0x7f2a8402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5344985b2bcbf750 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:22.733 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:51.941 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:22.735 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:51.943 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:22.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 1 -- 172.21.15.49:0/767518384 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a840317c0 con 0x7f2a940aa0f0 2024-05-15T13:10:22.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:22.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:22.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 10 monclient: dump: 2024-05-15T13:10:22.738 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:22.738 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.738 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.738 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.738 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 10 monclient: _finish_auth 0 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 10 monclient: _check_auth_tickets 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 1 -- 172.21.15.49:0/767518384 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a84037070 con 0x7f2a940aa0f0 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 1 -- 172.21.15.49:0/767518384 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84031b80 con 0x7f2a940aa0f0 2024-05-15T13:10:22.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a9b77e640 5 monclient: authenticate success, global_id 22051 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a9b77e640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a9b77e640 4 set_mon_vals no callback set 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.748+0000 7f2a8bfff640 10 monclient: dump: 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:22.740 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:22.741 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.749+0000 7f2a9b77e640 10 monclient: shutdown 2024-05-15T13:10:22.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.749+0000 7f2a9b77e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:22.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.749+0000 7f2a9b77e640 1 -- 172.21.15.49:0/767518384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a940aa0f0 msgr2=0x7f2a9413f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.749+0000 7f2a9b77e640 1 --2- 172.21.15.49:0/767518384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a940aa0f0 0x7f2a9413f790 secure :-1 s=READY pgs=5462 cs=0 l=1 rev1=1 crypto rx=0x7f2a84009870 tx=0x7f2a8402b860 comp rx=0 tx=0).stop 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.749+0000 7f2a9b77e640 1 -- 172.21.15.49:0/767518384 shutdown_connections 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.749+0000 7f2a9b77e640 1 --2- 172.21.15.49:0/767518384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a940aa0f0 0x7f2a9413f790 unknown :-1 s=CLOSED pgs=5462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.749+0000 7f2a9b77e640 1 --2- 172.21.15.49:0/767518384 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a940b5410 0x7f2a94145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.749+0000 7f2a9b77e640 1 -- 172.21.15.49:0/767518384 >> 172.21.15.49:0/767518384 conn(0x7f2a94066f00 msgr2=0x7f2a94067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.749+0000 7f2a9b77e640 1 -- 172.21.15.49:0/767518384 shutdown_connections 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.749+0000 7f2a9b77e640 1 -- 172.21.15.49:0/767518384 wait complete. 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.750+0000 7f2a9b77e640 10 monclient: build_initial_monmap 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.750+0000 7f2a9b77e640 10 monclient: monmap: 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:22.744 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.750+0000 7f2a9b77e640 1 librados: starting msgr at 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.750+0000 7f2a9b77e640 1 librados: starting objecter 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.750+0000 7f2a9b77e640 1 Processor -- start 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.750+0000 7f2a9b77e640 1 -- start start 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.750+0000 7f2a9b77e640 1 librados: setting wanted keys 2024-05-15T13:10:22.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.750+0000 7f2a9b77e640 1 librados: calling monclient init 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.750+0000 7f2a9b77e640 10 monclient: init 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a940aa0f0 0x7f2a941cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 10 monclient(hunting): picked mon.noname-b con 0x7f2a940aa0f0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a940b5410 0x7f2a941cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 10 monclient(hunting): picked mon.noname-a con 0x7f2a940b5410 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a9413fcd0 0x7f2a941cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 10 monclient(hunting): picked mon.noname-c con 0x7f2a9413fcd0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940a9bd0 con 0x7f2a940b5410 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940ab770 con 0x7f2a940aa0f0 2024-05-15T13:10:22.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940aa530 con 0x7f2a9413fcd0 2024-05-15T13:10:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.751+0000 7f2a9b77e640 10 monclient(hunting): authenticate will time out at 3780.912927s 2024-05-15T13:10:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.752+0000 7f2a99d1b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a9413fcd0 0x7f2a941cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.754+0000 7f2a99d1b640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a9413fcd0 0x7f2a941cf350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:37238/0 (socket says 172.21.15.49:37238) 2024-05-15T13:10:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.754+0000 7f2a99d1b640 1 -- 172.21.15.49:0/2790224016 learned_addr learned my addr 172.21.15.49:0/2790224016 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.754+0000 7f2a99d1b640 10 monclient(hunting): get_auth_request con 0x7f2a9413fcd0 auth_method 0 2024-05-15T13:10:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.754+0000 7f2a99d1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.754+0000 7f2a99d1b640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.754+0000 7f2a99d1b640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:22.748 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:38:51.911 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:22.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.755+0000 7f2a99d1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:22.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.755+0000 7f2a99d1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:22.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.755+0000 7f2a99d1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:22.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.755+0000 7f2a98d19640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a940b5410 0x7f2a941cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.757+0000 7f2a98d19640 10 monclient(hunting): get_auth_request con 0x7f2a940b5410 auth_method 0 2024-05-15T13:10:22.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.757+0000 7f2a98d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:22.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.757+0000 7f2a98d19640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.757+0000 7f2a98d19640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:22.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a98d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a98d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a98d19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a99d1b640 10 monclient(hunting): handle_auth_done global_id 17639 payload 824 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a99d1b640 1 -- 172.21.15.49:0/2790224016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a940aa0f0 msgr2=0x7f2a941cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a99d1b640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a940aa0f0 0x7f2a941cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a99d1b640 1 -- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a940b5410 msgr2=0x7f2a941cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a99d1b640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a940b5410 0x7f2a941cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a99d1b640 10 monclient: _finish_hunting 0 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a99d1b640 1 monclient: found mon.noname-c 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a99d1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a99d1b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T13:10:22.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.758+0000 7f2a99d1b640 1 -- 172.21.15.49:0/2790224016 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a941ce900 con 0x7f2a9413fcd0 2024-05-15T13:10:22.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.761+0000 7f2a99d1b640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a9413fcd0 0x7f2a941cf350 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00b970 tx=0x7f2a8c00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:22.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.761+0000 7f2a89ffb640 10 client.?.objecter ms_handle_connect 0x7f2a9413fcd0 2024-05-15T13:10:22.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.761+0000 7f2a89ffb640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:22.757 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:38:51.966 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:22.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 1 -- 172.21.15.49:0/2790224016 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c019070 con 0x7f2a9413fcd0 2024-05-15T13:10:22.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:22.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:22.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 10 monclient: dump: 2024-05-15T13:10:22.761 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:22.761 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.761 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.761 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 10 monclient: _finish_auth 0 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 10 monclient: _check_auth_tickets 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 1 -- 172.21.15.49:0/2790224016 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8c00cb40 con 0x7f2a9413fcd0 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 1 -- 172.21.15.49:0/2790224016 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c014440 con 0x7f2a9413fcd0 2024-05-15T13:10:22.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a89ffb640 10 monclient: dump: 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a9b77e640 5 monclient: authenticate success, global_id 17639 2024-05-15T13:10:22.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a9b77e640 10 monclient: _renew_subs 2024-05-15T13:10:22.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a9b77e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:22.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a941ceb70 con 0x7f2a9413fcd0 2024-05-15T13:10:22.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a8b7fe640 4 set_mon_vals no callback set 2024-05-15T13:10:22.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a9b77e640 10 client.17639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:22.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a9b77e640 10 monclient: _renew_subs 2024-05-15T13:10:22.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a9b77e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:22.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a941d7f80 con 0x7f2a9413fcd0 2024-05-15T13:10:22.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.771+0000 7f2a9b77e640 1 librados: init done 2024-05-15T13:10:22.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.772+0000 7f2a9b77e640 10 client.17639.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:22.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.772+0000 7f2a9b77e640 15 client.17639.objecter _session_command_op_assign -1 1 2024-05-15T13:10:22.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.772+0000 7f2a9b77e640 20 client.17639.objecter _get_session osd=-1 returning homeless 2024-05-15T13:10:22.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.772+0000 7f2a9b77e640 10 client.17639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:22.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.772+0000 7f2a9b77e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:22.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.772+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a9404f0e0 con 0x7f2a9413fcd0 2024-05-15T13:10:22.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.800+0000 7f2a89ffb640 1 -- 172.21.15.49:0/2790224016 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2a8c01c070 con 0x7f2a9413fcd0 2024-05-15T13:10:22.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.801+0000 7f2a89ffb640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2a7806ceb0 0x7f2a7806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.801+0000 7f2a9951a640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2a7806ceb0 0x7f2a7806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.801+0000 7f2a9951a640 10 monclient: get_auth_request con 0x7f2a7806ceb0 auth_method 0 2024-05-15T13:10:22.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.802+0000 7f2a89ffb640 10 client.17639.objecter ms_handle_connect 0x7f2a7806ceb0 2024-05-15T13:10:22.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.802+0000 7f2a9951a640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2a7806ceb0 0x7f2a7806f370 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009710 tx=0x7f2a90006c90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:22.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 1 -- 172.21.15.49:0/2790224016 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f2a8c056c50 con 0x7f2a9413fcd0 2024-05-15T13:10:22.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 10 client.17639.objecter ms_dispatch 0x7f2a94000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:22.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 3 client.17639.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:22.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 3 client.17639.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:22.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 10 client.17639.objecter checking command 1 2024-05-15T13:10:22.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] conn(0x7f2a780758a0 0x7f2a78077d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 20 client.17639.objecter _get_session s=0x7f2a780751d0 osd=4 3 2024-05-15T13:10:22.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 20 client.17639.objecter put_session s=0x7f2a780751d0 osd=4 3 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 15 client.17639.objecter _session_command_op_remove -1 1 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 20 client.17639.objecter _get_session s=0x7f2a780751d0 osd=4 3 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 20 client.17639.objecter get_session s=0x7f2a780751d0 osd=4 3 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 15 client.17639.objecter _session_command_op_assign 4 1 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 20 client.17639.objecter put_session s=0x7f2a780751d0 osd=4 4 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 10 client.17639.objecter _send_command 1 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 1 -- 172.21.15.49:0/2790224016 --> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a8c015e50 con 0x7f2a780758a0 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 20 client.17639.objecter dump_active .. 0 homeless 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 1 -- 172.21.15.49:0/2790224016 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_get_version_reply(handle=1 version=1251) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a8c054e50 con 0x7f2a9413fcd0 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.805+0000 7f2a89ffb640 10 monclient: handle_get_version_reply finishing 1 version 1251 2024-05-15T13:10:22.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.806+0000 7f2a98d19640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] conn(0x7f2a780758a0 0x7f2a78077d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.806+0000 7f2a98d19640 10 monclient: get_auth_request con 0x7f2a780758a0 auth_method 0 2024-05-15T13:10:22.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.808+0000 7f2a89ffb640 10 client.17639.objecter ms_handle_connect 0x7f2a780758a0 2024-05-15T13:10:22.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.808+0000 7f2a98d19640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] conn(0x7f2a780758a0 0x7f2a78077d40 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:22.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.833+0000 7f2a89ffb640 1 -- 172.21.15.49:0/2790224016 <== osd.4 v2:172.21.15.88:6816/3384996234 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2a8c015e50 con 0x7f2a780758a0 2024-05-15T13:10:22.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.833+0000 7f2a89ffb640 10 client.17639.objecter ms_dispatch 0x7f2a94000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:10:22.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.833+0000 7f2a89ffb640 10 client.17639.objecter _finish_command 1 = system:0 2024-05-15T13:10:22.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.833+0000 7f2a89ffb640 20 client.17639.objecter put_session s=0x7f2a780751d0 osd=4 4 2024-05-15T13:10:22.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.833+0000 7f2a89ffb640 15 client.17639.objecter _session_command_op_remove 4 1 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 10 client.17639.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 15 client.17639.objecter _session_command_op_assign -1 2 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 20 client.17639.objecter _get_session s=0x7f2a780751d0 osd=4 3 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 20 client.17639.objecter put_session s=0x7f2a780751d0 osd=4 3 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 20 client.17639.objecter _get_session s=0x7f2a780751d0 osd=4 3 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 15 client.17639.objecter _session_command_op_remove -1 2 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 20 client.17639.objecter get_session s=0x7f2a780751d0 osd=4 3 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 15 client.17639.objecter _session_command_op_assign 4 2 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 20 client.17639.objecter put_session s=0x7f2a780751d0 osd=4 4 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 10 client.17639.objecter _send_command 2 2024-05-15T13:10:22.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.846+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 --> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2a941cd5e0 con 0x7f2a780758a0 2024-05-15T13:10:22.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.859+0000 7f2a89ffb640 1 -- 172.21.15.49:0/2790224016 <== osd.4 v2:172.21.15.88:6816/3384996234 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2a941cd5e0 con 0x7f2a780758a0 2024-05-15T13:10:22.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.859+0000 7f2a89ffb640 10 client.17639.objecter ms_dispatch 0x7f2a94000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:10:22.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.859+0000 7f2a89ffb640 10 client.17639.objecter _finish_command 2 = system:0 2024-05-15T13:10:22.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.859+0000 7f2a89ffb640 20 client.17639.objecter put_session s=0x7f2a780751d0 osd=4 4 2024-05-15T13:10:22.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.859+0000 7f2a89ffb640 15 client.17639.objecter _session_command_op_remove 4 2 2024-05-15T13:10:22.850 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-05-15T13:10:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:10:22.850 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-05-15T13:10:22.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 10 librados: watch_flush enter 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 10 librados: watch_flush exit 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 10 client.17639.objecter close_session for osd.4 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 >> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] conn(0x7f2a780758a0 msgr2=0x7f2a78077d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.88:6816/3384996234,v1:172.21.15.88:6817/3384996234] conn(0x7f2a780758a0 0x7f2a78077d40 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 20 client.17639.objecter put_session s=0x7f2a780751d0 osd=4 1 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 20 client.17639.objecter shutdown clearing up homeless session... 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 10 client.17639.objecter successfully canceled tick 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2a7806ceb0 msgr2=0x7f2a7806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2a7806ceb0 0x7f2a7806f370 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009710 tx=0x7f2a90006c90 comp rx=0 tx=0).stop 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 10 monclient: shutdown 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a9413fcd0 msgr2=0x7f2a941cf350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:22.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 --2- 172.21.15.49:0/2790224016 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a9413fcd0 0x7f2a941cf350 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00b970 tx=0x7f2a8c00be30 comp rx=0 tx=0).stop 2024-05-15T13:10:22.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a99d1b640 1 -- 172.21.15.49:0/2790224016 reap_dead start 2024-05-15T13:10:22.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 shutdown_connections 2024-05-15T13:10:22.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 >> 172.21.15.49:0/2790224016 conn(0x7f2a94066f00 msgr2=0x7f2a940b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:22.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 shutdown_connections 2024-05-15T13:10:22.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 -- 172.21.15.49:0/2790224016 wait complete. 2024-05-15T13:10:22.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.860+0000 7f2a9b77e640 1 librados: shutdown 2024-05-15T13:10:22.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-15T13:10:22.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.979+0000 7fe0d833a640 10 monclient: get_monmap_and_config 2024-05-15T13:10:22.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.979+0000 7fe0d833a640 10 monclient: build_initial_monmap 2024-05-15T13:10:22.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.979+0000 7fe0d833a640 10 monclient: monmap: 2024-05-15T13:10:22.969 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:22.969 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:22.969 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:22.980256+0000 2024-05-15T13:10:22.969 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:22.980256+0000 2024-05-15T13:10:22.970 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:22.970 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:22.970 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:22.970 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:22.970 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:22.970 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:22.970 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:22.970 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:22.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.980+0000 7fe0d833a640 1 Processor -- start 2024-05-15T13:10:22.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.980+0000 7fe0d833a640 1 -- start start 2024-05-15T13:10:22.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.980+0000 7fe0d833a640 10 monclient: init 2024-05-15T13:10:22.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:22.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0d00b5410 0x7fe0d00a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 10 monclient(hunting): picked mon.noname-b con 0x7fe0d00b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0d013f990 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe0d0145c00 0x7fe0d0146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 10 monclient(hunting): picked mon.noname-c con 0x7fe0d0145c00 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d00ab470 con 0x7fe0d013f990 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d00b57f0 con 0x7fe0d0145c00 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d00a9ea0 con 0x7fe0d00b5410 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.981+0000 7fe0d833a640 10 monclient(hunting): authenticate will time out at 3781.142799s 2024-05-15T13:10:22.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.986+0000 7fe0d60d6640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0d00b5410 0x7fe0d00a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:22.997+0000 7fe0d68d7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe0d0145c00 0x7fe0d0146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:22.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.002+0000 7fe0d58d5640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe0d013f990 0x7fe0d013fdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:55252/0 (socket says 172.21.15.49:55252) 2024-05-15T13:10:22.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.002+0000 7fe0d58d5640 1 -- 172.21.15.49:0/3619097841 learned_addr learned my addr 172.21.15.49:0/3619097841 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:22.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.003+0000 7fe0d68d7640 10 monclient(hunting): get_auth_request con 0x7fe0d0145c00 auth_method 0 2024-05-15T13:10:22.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.003+0000 7fe0d68d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:22.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.003+0000 7fe0d68d7640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:22.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.003+0000 7fe0d68d7640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:22.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.004+0000 7fe0d68d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:22.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.004+0000 7fe0d68d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:22.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.004+0000 7fe0d68d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:23.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.024+0000 7fe0d68d7640 10 monclient(hunting): handle_auth_done global_id 22054 payload 274 2024-05-15T13:10:23.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.024+0000 7fe0d68d7640 1 -- 172.21.15.49:0/3619097841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0d00b5410 msgr2=0x7fe0d00a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:23.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.024+0000 7fe0d68d7640 1 --2- 172.21.15.49:0/3619097841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0d00b5410 0x7fe0d00a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.024+0000 7fe0d68d7640 1 -- 172.21.15.49:0/3619097841 >> v1:172.21.15.88:6789/0 conn(0x7fe0d013f990 legacy=0x7fe0d013fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.024+0000 7fe0d68d7640 10 monclient: _finish_hunting 0 2024-05-15T13:10:23.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.024+0000 7fe0d68d7640 1 monclient: found mon.noname-c 2024-05-15T13:10:23.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.024+0000 7fe0d68d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:23.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.024+0000 7fe0d68d7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T13:10:23.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.024+0000 7fe0d68d7640 1 -- 172.21.15.49:0/3619097841 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d00aa0c0 con 0x7fe0d0145c00 2024-05-15T13:10:23.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.058+0000 7fe0d68d7640 1 --2- 172.21.15.49:0/3619097841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d0145c00 0x7fe0d0146070 secure :-1 s=READY pgs=5464 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009dc0 tx=0x7fe0c802be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25ff70eab29f4d8f server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.154+0000 7fe0d50d4640 1 -- 172.21.15.49:0/3619097841 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8039070 con 0x7fe0d0145c00 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 10 monclient: dump: 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:23.145 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 10 monclient: _finish_auth 0 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 10 monclient: _check_auth_tickets 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 1 -- 172.21.15.49:0/3619097841 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c802cbb0 con 0x7fe0d0145c00 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 5 monclient: authenticate success, global_id 22054 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 1 -- 172.21.15.49:0/3619097841 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8034440 con 0x7fe0d0145c00 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 4 set_mon_vals no callback set 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d50d4640 10 monclient: dump: 2024-05-15T13:10:23.147 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:23.147 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:23.147 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.147 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.147 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:23.147 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.147 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:23.147 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:23.147 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:23.147 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 10 monclient: shutdown 2024-05-15T13:10:23.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:23.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 1 -- 172.21.15.49:0/3619097841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d0145c00 msgr2=0x7fe0d0146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 1 --2- 172.21.15.49:0/3619097841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d0145c00 0x7fe0d0146070 secure :-1 s=READY pgs=5464 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009dc0 tx=0x7fe0c802be00 comp rx=0 tx=0).stop 2024-05-15T13:10:23.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 1 -- 172.21.15.49:0/3619097841 shutdown_connections 2024-05-15T13:10:23.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 1 --2- 172.21.15.49:0/3619097841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d0145c00 0x7fe0d0146070 unknown :-1 s=CLOSED pgs=5464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 1 --2- 172.21.15.49:0/3619097841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0d00b5410 0x7fe0d00a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.155+0000 7fe0d833a640 1 -- 172.21.15.49:0/3619097841 >> 172.21.15.49:0/3619097841 conn(0x7fe0d0066f00 msgr2=0x7fe0d0067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.156+0000 7fe0d833a640 1 -- 172.21.15.49:0/3619097841 shutdown_connections 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.156+0000 7fe0d833a640 1 -- 172.21.15.49:0/3619097841 wait complete. 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.156+0000 7fe0d833a640 10 monclient: build_initial_monmap 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.156+0000 7fe0d833a640 10 monclient: monmap: 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.156+0000 7fe0d833a640 1 librados: starting msgr at 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.156+0000 7fe0d833a640 1 librados: starting objecter 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 1 Processor -- start 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 1 -- start start 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 1 librados: setting wanted keys 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 1 librados: calling monclient init 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient: init 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0d00b5410 0x7fe0d0143100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient(hunting): picked mon.noname-a con 0x7fe0d00b5410 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d013f990 0x7fe0d01410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient(hunting): picked mon.noname-b con 0x7fe0d013f990 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0d0145c00 0x7fe0d0136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient(hunting): picked mon.noname-c con 0x7fe0d0145c00 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d00b4aa0 con 0x7fe0d00b5410 2024-05-15T13:10:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d00b4920 con 0x7fe0d013f990 2024-05-15T13:10:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d014cf30 con 0x7fe0d0145c00 2024-05-15T13:10:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.157+0000 7fe0d833a640 10 monclient(hunting): authenticate will time out at 3781.319124s 2024-05-15T13:10:23.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.163+0000 7fe0d58d5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d013f990 0x7fe0d01410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:23.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.164+0000 7fe0d58d5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d013f990 0x7fe0d01410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43240/0 (socket says 172.21.15.49:43240) 2024-05-15T13:10:23.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.164+0000 7fe0d58d5640 1 -- 172.21.15.49:0/3932872538 learned_addr learned my addr 172.21.15.49:0/3932872538 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:23.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.164+0000 7fe0d58d5640 10 monclient(hunting): get_auth_request con 0x7fe0d013f990 auth_method 0 2024-05-15T13:10:23.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.164+0000 7fe0d58d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:23.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.164+0000 7fe0d58d5640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:23.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.164+0000 7fe0d58d5640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:23.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.164+0000 7fe0d60d6640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0d00b5410 0x7fe0d0143100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:23.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.165+0000 7fe0d58d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:23.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.165+0000 7fe0d58d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:23.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.165+0000 7fe0d58d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:23.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.168+0000 7fe0d58d5640 10 monclient(hunting): handle_auth_done global_id 22057 payload 824 2024-05-15T13:10:23.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.168+0000 7fe0d58d5640 1 -- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0d0145c00 msgr2=0x7fe0d0136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.168+0000 7fe0d58d5640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0d0145c00 0x7fe0d0136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.168+0000 7fe0d58d5640 1 -- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0d00b5410 msgr2=0x7fe0d0143100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.168+0000 7fe0d58d5640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0d00b5410 0x7fe0d0143100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.168+0000 7fe0d58d5640 10 monclient: _finish_hunting 0 2024-05-15T13:10:23.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.168+0000 7fe0d58d5640 1 monclient: found mon.noname-b 2024-05-15T13:10:23.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.168+0000 7fe0d58d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:23.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.168+0000 7fe0d58d5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:23.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.168+0000 7fe0d58d5640 1 -- 172.21.15.49:0/3932872538 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d0142690 con 0x7fe0d013f990 2024-05-15T13:10:23.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.171+0000 7fe0d58d5640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d013f990 0x7fe0d01410d0 secure :-1 s=READY pgs=5465 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000c400 tx=0x7fe0c000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:23.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.171+0000 7fe0c6ffd640 10 client.?.objecter ms_handle_connect 0x7fe0d013f990 2024-05-15T13:10:23.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.172+0000 7fe0c6ffd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:23.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 1 -- 172.21.15.49:0/3932872538 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c0017070 con 0x7fe0d013f990 2024-05-15T13:10:23.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:23.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:23.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 10 monclient: dump: 2024-05-15T13:10:23.175 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:23.175 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:23.175 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.175 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.175 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 10 monclient: _finish_auth 0 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 10 monclient: _check_auth_tickets 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 1 -- 172.21.15.49:0/3932872538 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c0005ce0 con 0x7fe0d013f990 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 1 -- 172.21.15.49:0/3932872538 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c00072f0 con 0x7fe0d013f990 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0c6ffd640 10 monclient: dump: 2024-05-15T13:10:23.176 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:23.177 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:23.177 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.177 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.177 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:23.177 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.177 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:23.177 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:23.177 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:23.177 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0d833a640 5 monclient: authenticate success, global_id 22057 2024-05-15T13:10:23.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0d833a640 10 monclient: _renew_subs 2024-05-15T13:10:23.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0d833a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:23.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0d833a640 1 -- 172.21.15.49:0/3932872538 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d0142970 con 0x7fe0d013f990 2024-05-15T13:10:23.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0d48d3640 4 set_mon_vals no callback set 2024-05-15T13:10:23.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0d833a640 10 client.22057.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:23.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0d833a640 10 monclient: _renew_subs 2024-05-15T13:10:23.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0d833a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:23.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0d833a640 1 -- 172.21.15.49:0/3932872538 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d01416c0 con 0x7fe0d013f990 2024-05-15T13:10:23.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.185+0000 7fe0d833a640 1 librados: init done 2024-05-15T13:10:23.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.186+0000 7fe0d833a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:23.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.186+0000 7fe0d833a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:23.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.186+0000 7fe0d833a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:23.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.186+0000 7fe0d833a640 1 -- 172.21.15.49:0/3932872538 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d01409b0 con 0x7fe0d013f990 2024-05-15T13:10:23.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.444+0000 7fe0c6ffd640 1 -- 172.21.15.49:0/3932872538 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe0c001a070 con 0x7fe0d013f990 2024-05-15T13:10:23.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.445+0000 7fe0c6ffd640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe0b406ceb0 0x7fe0b406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.445+0000 7fe0d60d6640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe0b406ceb0 0x7fe0b406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:23.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.445+0000 7fe0d60d6640 10 monclient: get_auth_request con 0x7fe0b406ceb0 auth_method 0 2024-05-15T13:10:23.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.445+0000 7fe0c6ffd640 10 client.22057.objecter ms_handle_connect 0x7fe0b406ceb0 2024-05-15T13:10:23.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.445+0000 7fe0d60d6640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe0b406ceb0 0x7fe0b406f370 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc006fd0 tx=0x7fe0cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:23.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.448+0000 7fe0c6ffd640 1 -- 172.21.15.49:0/3932872538 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fe0c00562e0 con 0x7fe0d013f990 2024-05-15T13:10:23.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.448+0000 7fe0c6ffd640 10 client.22057.objecter ms_dispatch 0x7fe0d0000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:23.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.448+0000 7fe0c6ffd640 3 client.22057.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:23.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.448+0000 7fe0c6ffd640 3 client.22057.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:23.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.448+0000 7fe0c6ffd640 20 client.22057.objecter dump_active .. 0 homeless 2024-05-15T13:10:23.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.458+0000 7fe0c6ffd640 1 -- 172.21.15.49:0/3932872538 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe0c0021020 con 0x7fe0d013f990 2024-05-15T13:10:23.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.458+0000 7fe0c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:23.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.458+0000 7fe0c6ffd640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:23.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.653+0000 7fe0d833a640 1 -- 172.21.15.49:0/3932872538 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0d0146ad0 con 0x7fe0b406ceb0 2024-05-15T13:10:23.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.653+0000 7fe0c6ffd640 1 -- 172.21.15.49:0/3932872538 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe0d0146ad0 con 0x7fe0b406ceb0 2024-05-15T13:10:23.643 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 4 to deep-scrub 2024-05-15T13:10:23.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.655+0000 7fe0d833a640 10 librados: watch_flush enter 2024-05-15T13:10:23.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.655+0000 7fe0d833a640 10 librados: watch_flush exit 2024-05-15T13:10:23.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.655+0000 7fe0d833a640 20 client.22057.objecter shutdown clearing up homeless session... 2024-05-15T13:10:23.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.655+0000 7fe0d833a640 10 client.22057.objecter successfully canceled tick 2024-05-15T13:10:23.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 -- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe0b406ceb0 msgr2=0x7fe0b406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe0b406ceb0 0x7fe0b406f370 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc006fd0 tx=0x7fe0cc008040 comp rx=0 tx=0).stop 2024-05-15T13:10:23.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 10 monclient: shutdown 2024-05-15T13:10:23.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:23.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 -- 172.21.15.49:0/3932872538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d013f990 msgr2=0x7fe0d01410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d013f990 0x7fe0d01410d0 secure :-1 s=READY pgs=5465 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000c400 tx=0x7fe0c000c8c0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 -- 172.21.15.49:0/3932872538 shutdown_connections 2024-05-15T13:10:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0d0145c00 0x7fe0d0136bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe0b406ceb0 0x7fe0b406f370 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0d013f990 0x7fe0d01410d0 unknown :-1 s=CLOSED pgs=5465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 --2- 172.21.15.49:0/3932872538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0d00b5410 0x7fe0d0143100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 -- 172.21.15.49:0/3932872538 >> 172.21.15.49:0/3932872538 conn(0x7fe0d0066f00 msgr2=0x7fe0d00b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 -- 172.21.15.49:0/3932872538 shutdown_connections 2024-05-15T13:10:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 -- 172.21.15.49:0/3932872538 wait complete. 2024-05-15T13:10:23.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.656+0000 7fe0d833a640 1 librados: shutdown 2024-05-15T13:10:23.658 INFO:tasks.ceph:Scrubbing osd.5 2024-05-15T13:10:23.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.775+0000 7fe14c97d640 10 monclient: get_monmap_and_config 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.775+0000 7fe14c97d640 10 monclient: build_initial_monmap 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.775+0000 7fe14c97d640 10 monclient: monmap: 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:23.775741+0000 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:23.775741+0000 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:23.765 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:23.766 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:23.766 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:23.766 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 1 Processor -- start 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 1 -- start start 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 10 monclient: init 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe1480adb10 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe1480b04d0 0x7fe1481427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 10 monclient(hunting): picked mon.noname-c con 0x7fe1480b04d0 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe14806ef50 0x7fe148142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 10 monclient(hunting): picked mon.noname-a con 0x7fe14806ef50 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1480b0950 con 0x7fe1480adb10 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:23.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe148070240 con 0x7fe14806ef50 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1480ae1e0 con 0x7fe1480b04d0 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.776+0000 7fe14c97d640 10 monclient(hunting): authenticate will time out at 3781.938163s 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.777+0000 7fe13ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe1480b04d0 0x7fe1481427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.777+0000 7fe146d9d640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe14806ef50 0x7fe148142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.777+0000 7fe14659c640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fe1480adb10 0x7fe14813f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:55276/0 (socket says 172.21.15.49:55276) 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.777+0000 7fe14659c640 1 -- 172.21.15.49:0/2156825162 learned_addr learned my addr 172.21.15.49:0/2156825162 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.777+0000 7fe13ffff640 1 --2- 172.21.15.49:0/2156825162 >> v2:172.21.15.88:3300/0 conn(0x7fe1480b04d0 0x7fe1481427f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43252/0 (socket says 172.21.15.49:43252) 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.777+0000 7fe13ffff640 10 monclient(hunting): get_auth_request con 0x7fe1480b04d0 auth_method 0 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.777+0000 7fe13ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.777+0000 7fe13ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.777+0000 7fe13ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:23.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.778+0000 7fe13ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:23.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.778+0000 7fe13ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:23.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.778+0000 7fe13ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:23.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.778+0000 7fe146d9d640 10 monclient(hunting): get_auth_request con 0x7fe14806ef50 auth_method 0 2024-05-15T13:10:23.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.778+0000 7fe146d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:23.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.778+0000 7fe146d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:23.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.778+0000 7fe146d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.778+0000 7fe146d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.778+0000 7fe146d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.778+0000 7fe146d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.779+0000 7fe13ffff640 10 monclient(hunting): handle_auth_done global_id 22060 payload 274 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.779+0000 7fe13ffff640 1 -- 172.21.15.49:0/2156825162 >> v2:172.21.15.49:3300/0 conn(0x7fe14806ef50 msgr2=0x7fe148142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.779+0000 7fe13ffff640 1 --2- 172.21.15.49:0/2156825162 >> v2:172.21.15.49:3300/0 conn(0x7fe14806ef50 0x7fe148142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.779+0000 7fe13ffff640 1 -- 172.21.15.49:0/2156825162 >> v1:172.21.15.49:6789/0 conn(0x7fe1480adb10 legacy=0x7fe14813f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.779+0000 7fe13ffff640 10 monclient: _finish_hunting 0 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.779+0000 7fe13ffff640 1 monclient: found mon.noname-c 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.779+0000 7fe13ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.779+0000 7fe13ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T13:10:23.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.779+0000 7fe13ffff640 1 -- 172.21.15.49:0/2156825162 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe148143270 con 0x7fe1480b04d0 2024-05-15T13:10:23.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.781+0000 7fe13ffff640 1 --2- 172.21.15.49:0/2156825162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1480b04d0 0x7fe1481427f0 secure :-1 s=READY pgs=5466 cs=0 l=1 rev1=1 crypto rx=0x7fe1300097a0 tx=0x7fe13002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a29dfcd05a0c381e server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:23.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 1 -- 172.21.15.49:0/2156825162 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1300315a0 con 0x7fe1480b04d0 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 10 monclient: dump: 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 10 monclient: _finish_auth 0 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 10 monclient: _check_auth_tickets 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 1 -- 172.21.15.49:0/2156825162 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe130037070 con 0x7fe1480b04d0 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe14c97d640 5 monclient: authenticate success, global_id 22060 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe14c97d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 1 -- 172.21.15.49:0/2156825162 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe130031960 con 0x7fe1480b04d0 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe145d9b640 10 monclient: dump: 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:23.777 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe14c97d640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.785+0000 7fe14c97d640 4 set_mon_vals no callback set 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.786+0000 7fe14c97d640 10 monclient: shutdown 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.786+0000 7fe14c97d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.786+0000 7fe14c97d640 1 -- 172.21.15.49:0/2156825162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1480b04d0 msgr2=0x7fe1481427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.786+0000 7fe14c97d640 1 --2- 172.21.15.49:0/2156825162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1480b04d0 0x7fe1481427f0 secure :-1 s=READY pgs=5466 cs=0 l=1 rev1=1 crypto rx=0x7fe1300097a0 tx=0x7fe13002b840 comp rx=0 tx=0).stop 2024-05-15T13:10:23.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.786+0000 7fe14c97d640 1 -- 172.21.15.49:0/2156825162 shutdown_connections 2024-05-15T13:10:23.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.786+0000 7fe14c97d640 1 --2- 172.21.15.49:0/2156825162 >> v2:172.21.15.49:3300/0 conn(0x7fe14806ef50 0x7fe148142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.786+0000 7fe14c97d640 1 --2- 172.21.15.49:0/2156825162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1480b04d0 0x7fe1481427f0 unknown :-1 s=CLOSED pgs=5466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.786+0000 7fe14c97d640 1 -- 172.21.15.49:0/2156825162 >> 172.21.15.49:0/2156825162 conn(0x7fe1480b45f0 msgr2=0x7fe1480b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:23.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.786+0000 7fe14c97d640 1 -- 172.21.15.49:0/2156825162 shutdown_connections 2024-05-15T13:10:23.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.786+0000 7fe14c97d640 1 -- 172.21.15.49:0/2156825162 wait complete. 2024-05-15T13:10:23.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 10 monclient: build_initial_monmap 2024-05-15T13:10:23.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 10 monclient: monmap: 2024-05-15T13:10:23.782 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:23.782 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:23.782 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:23.782 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:23.782 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:23.782 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 1 librados: starting msgr at 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 1 librados: starting objecter 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 1 Processor -- start 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 1 -- start start 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 1 librados: setting wanted keys 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 1 librados: calling monclient init 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 10 monclient: init 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.787+0000 7fe14c97d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe14806ef50 0x7fe14813a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 10 monclient(hunting): picked mon.noname-b con 0x7fe14806ef50 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1480adb10 0x7fe1481db2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 10 monclient(hunting): picked mon.noname-a con 0x7fe1480adb10 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1480b04d0 0x7fe148145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 10 monclient(hunting): picked mon.noname-c con 0x7fe1480b04d0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1480ae360 con 0x7fe1480adb10 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14813ff60 con 0x7fe14806ef50 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14814cf00 con 0x7fe1480b04d0 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14c97d640 10 monclient(hunting): authenticate will time out at 3781.949494s 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe13ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1480adb10 0x7fe1481db2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe14806ef50 0x7fe14813a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe146d9d640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1480b04d0 0x7fe148145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe13ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1480adb10 0x7fe1481db2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60014/0 (socket says 172.21.15.49:60014) 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe13ffff640 1 -- 172.21.15.49:0/4166449187 learned_addr learned my addr 172.21.15.49:0/4166449187 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe13ffff640 10 monclient(hunting): get_auth_request con 0x7fe1480adb10 auth_method 0 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe13ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe13ffff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe13ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14659c640 10 monclient(hunting): get_auth_request con 0x7fe14806ef50 auth_method 0 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14659c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.788+0000 7fe14659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe146d9d640 10 monclient(hunting): get_auth_request con 0x7fe1480b04d0 auth_method 0 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe146d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe146d9d640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe146d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe14659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:23.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe14659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe14659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe13ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe13ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe13ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe146d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe146d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.789+0000 7fe146d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.791+0000 7fe14659c640 10 monclient(hunting): handle_auth_done global_id 22063 payload 824 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.791+0000 7fe14659c640 1 -- 172.21.15.49:0/4166449187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1480b04d0 msgr2=0x7fe148145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.792+0000 7fe14659c640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1480b04d0 0x7fe148145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.792+0000 7fe14659c640 1 -- 172.21.15.49:0/4166449187 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1480adb10 msgr2=0x7fe1481db2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.792+0000 7fe14659c640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1480adb10 0x7fe1481db2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.792+0000 7fe14659c640 10 monclient: _finish_hunting 0 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.792+0000 7fe14659c640 1 monclient: found mon.noname-b 2024-05-15T13:10:23.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.792+0000 7fe14659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:23.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.792+0000 7fe14659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:23.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.792+0000 7fe14659c640 1 -- 172.21.15.49:0/4166449187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1481442b0 con 0x7fe14806ef50 2024-05-15T13:10:23.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.794+0000 7fe14659c640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe14806ef50 0x7fe14813a3e0 secure :-1 s=READY pgs=5467 cs=0 l=1 rev1=1 crypto rx=0x7fe138000f60 tx=0x7fe13800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:23.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.794+0000 7fe13effd640 10 client.?.objecter ms_handle_connect 0x7fe14806ef50 2024-05-15T13:10:23.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.794+0000 7fe13effd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 1 -- 172.21.15.49:0/4166449187 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138017070 con 0x7fe14806ef50 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 10 monclient: dump: 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:23.789 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 10 monclient: _finish_auth 0 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 10 monclient: _check_auth_tickets 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 1 -- 172.21.15.49:0/4166449187 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe138005ce0 con 0x7fe14806ef50 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 1 -- 172.21.15.49:0/4166449187 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138007310 con 0x7fe14806ef50 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe14c97d640 5 monclient: authenticate success, global_id 22063 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe14c97d640 10 monclient: _renew_subs 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe14c97d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe148144590 con 0x7fe14806ef50 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe13effd640 10 monclient: dump: 2024-05-15T13:10:23.790 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe14559a640 4 set_mon_vals no callback set 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe14c97d640 10 client.22063.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe14c97d640 10 monclient: _renew_subs 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe14c97d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe148144fe0 con 0x7fe14806ef50 2024-05-15T13:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.797+0000 7fe14c97d640 1 librados: init done 2024-05-15T13:10:23.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.798+0000 7fe14c97d640 10 client.22063.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:23.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.798+0000 7fe14c97d640 15 client.22063.objecter _session_command_op_assign -1 1 2024-05-15T13:10:23.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.798+0000 7fe14c97d640 20 client.22063.objecter _get_session osd=-1 returning homeless 2024-05-15T13:10:23.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.798+0000 7fe14c97d640 10 client.22063.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:23.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.798+0000 7fe14c97d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:23.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.798+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe148145220 con 0x7fe14806ef50 2024-05-15T13:10:23.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.817+0000 7fe13effd640 1 -- 172.21.15.49:0/4166449187 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe13801a070 con 0x7fe14806ef50 2024-05-15T13:10:23.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.818+0000 7fe13effd640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe12806ceb0 0x7fe12806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.818+0000 7fe13ffff640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe12806ceb0 0x7fe12806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:23.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.818+0000 7fe13ffff640 10 monclient: get_auth_request con 0x7fe12806ceb0 auth_method 0 2024-05-15T13:10:23.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.818+0000 7fe13effd640 10 client.22063.objecter ms_handle_connect 0x7fe12806ceb0 2024-05-15T13:10:23.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.818+0000 7fe13ffff640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe12806ceb0 0x7fe12806f370 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe130000c00 tx=0x7fe13003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:23.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 1 -- 172.21.15.49:0/4166449187 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fe138056520 con 0x7fe14806ef50 2024-05-15T13:10:23.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 10 client.22063.objecter ms_dispatch 0x7fe148000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:23.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 3 client.22063.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:23.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 3 client.22063.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:23.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 10 client.22063.objecter checking command 1 2024-05-15T13:10:23.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] conn(0x7fe1280758a0 0x7fe128077d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 20 client.22063.objecter _get_session s=0x7fe1280751d0 osd=5 3 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 20 client.22063.objecter put_session s=0x7fe1280751d0 osd=5 3 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 15 client.22063.objecter _session_command_op_remove -1 1 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 20 client.22063.objecter _get_session s=0x7fe1280751d0 osd=5 3 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 20 client.22063.objecter get_session s=0x7fe1280751d0 osd=5 3 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 15 client.22063.objecter _session_command_op_assign 5 1 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 20 client.22063.objecter put_session s=0x7fe1280751d0 osd=5 4 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 10 client.22063.objecter _send_command 1 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 1 -- 172.21.15.49:0/4166449187 --> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe138013e50 con 0x7fe1280758a0 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 20 client.22063.objecter dump_active .. 0 homeless 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 1 -- 172.21.15.49:0/4166449187 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=1251) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe138054720 con 0x7fe14806ef50 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe13effd640 10 monclient: handle_get_version_reply finishing 1 version 1251 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.820+0000 7fe146d9d640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] conn(0x7fe1280758a0 0x7fe128077d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:23.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.821+0000 7fe146d9d640 10 monclient: get_auth_request con 0x7fe1280758a0 auth_method 0 2024-05-15T13:10:23.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.823+0000 7fe146d9d640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] conn(0x7fe1280758a0 0x7fe128077d40 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:23.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.824+0000 7fe13effd640 10 client.22063.objecter ms_handle_connect 0x7fe1280758a0 2024-05-15T13:10:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.847+0000 7fe13effd640 1 -- 172.21.15.49:0/4166449187 <== osd.5 v2:172.21.15.88:6808/3085185190 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe138013e50 con 0x7fe1280758a0 2024-05-15T13:10:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.847+0000 7fe13effd640 10 client.22063.objecter ms_dispatch 0x7fe148000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:10:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.847+0000 7fe13effd640 10 client.22063.objecter _finish_command 1 = system:0 2024-05-15T13:10:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.847+0000 7fe13effd640 20 client.22063.objecter put_session s=0x7fe1280751d0 osd=5 4 2024-05-15T13:10:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.847+0000 7fe13effd640 15 client.22063.objecter _session_command_op_remove 5 1 2024-05-15T13:10:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 10 client.22063.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:10:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 15 client.22063.objecter _session_command_op_assign -1 2 2024-05-15T13:10:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 20 client.22063.objecter _get_session s=0x7fe1280751d0 osd=5 3 2024-05-15T13:10:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 20 client.22063.objecter put_session s=0x7fe1280751d0 osd=5 3 2024-05-15T13:10:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 20 client.22063.objecter _get_session s=0x7fe1280751d0 osd=5 3 2024-05-15T13:10:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 15 client.22063.objecter _session_command_op_remove -1 2 2024-05-15T13:10:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 20 client.22063.objecter get_session s=0x7fe1280751d0 osd=5 3 2024-05-15T13:10:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 15 client.22063.objecter _session_command_op_assign 5 2 2024-05-15T13:10:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 20 client.22063.objecter put_session s=0x7fe1280751d0 osd=5 4 2024-05-15T13:10:23.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 10 client.22063.objecter _send_command 2 2024-05-15T13:10:23.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.860+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 --> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe14804f0e0 con 0x7fe1280758a0 2024-05-15T13:10:23.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe13effd640 1 -- 172.21.15.49:0/4166449187 <== osd.5 v2:172.21.15.88:6808/3085185190 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe14804f0e0 con 0x7fe1280758a0 2024-05-15T13:10:23.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe13effd640 10 client.22063.objecter ms_dispatch 0x7fe148000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:10:23.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe13effd640 10 client.22063.objecter _finish_command 2 = system:0 2024-05-15T13:10:23.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe13effd640 20 client.22063.objecter put_session s=0x7fe1280751d0 osd=5 4 2024-05-15T13:10:23.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe13effd640 15 client.22063.objecter _session_command_op_remove 5 2 2024-05-15T13:10:23.863 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-05-15T13:10:23.863 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:10:23.864 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe14c97d640 10 librados: watch_flush enter 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe14c97d640 10 librados: watch_flush exit 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe14c97d640 10 client.22063.objecter close_session for osd.5 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 >> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] conn(0x7fe1280758a0 msgr2=0x7fe128077d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe14c97d640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.88:6808/3085185190,v1:172.21.15.88:6809/3085185190] conn(0x7fe1280758a0 0x7fe128077d40 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe14c97d640 20 client.22063.objecter put_session s=0x7fe1280751d0 osd=5 1 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe14c97d640 20 client.22063.objecter shutdown clearing up homeless session... 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe14c97d640 10 client.22063.objecter successfully canceled tick 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe12806ceb0 msgr2=0x7fe12806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.873+0000 7fe14c97d640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fe12806ceb0 0x7fe12806f370 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe130000c00 tx=0x7fe13003d040 comp rx=0 tx=0).stop 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.874+0000 7fe14c97d640 10 monclient: shutdown 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.874+0000 7fe14c97d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:23.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.874+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe14806ef50 msgr2=0x7fe14813a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:23.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.874+0000 7fe14c97d640 1 --2- 172.21.15.49:0/4166449187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe14806ef50 0x7fe14813a3e0 secure :-1 s=READY pgs=5467 cs=0 l=1 rev1=1 crypto rx=0x7fe138000f60 tx=0x7fe13800c800 comp rx=0 tx=0).stop 2024-05-15T13:10:23.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.874+0000 7fe146d9d640 1 -- 172.21.15.49:0/4166449187 reap_dead start 2024-05-15T13:10:23.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.874+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 shutdown_connections 2024-05-15T13:10:23.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.874+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 >> 172.21.15.49:0/4166449187 conn(0x7fe1480b45f0 msgr2=0x7fe1480b1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:23.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.874+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 shutdown_connections 2024-05-15T13:10:23.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.874+0000 7fe14c97d640 1 -- 172.21.15.49:0/4166449187 wait complete. 2024-05-15T13:10:23.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:23.874+0000 7fe14c97d640 1 librados: shutdown 2024-05-15T13:10:23.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-15T13:10:24.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.014+0000 7f126c71c640 10 monclient: get_monmap_and_config 2024-05-15T13:10:24.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.014+0000 7f126c71c640 10 monclient: build_initial_monmap 2024-05-15T13:10:24.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.014+0000 7f126c71c640 10 monclient: monmap: 2024-05-15T13:10:24.004 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:24.004 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.004 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:24.014677+0000 2024-05-15T13:10:24.004 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:24.014677+0000 2024-05-15T13:10:24.004 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:24.005 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.005 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:24.005 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:24.005 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:24.005 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:24.005 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:24.005 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.014+0000 7f126c71c640 1 Processor -- start 2024-05-15T13:10:24.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.014+0000 7f126c71c640 1 -- start start 2024-05-15T13:10:24.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.014+0000 7f126c71c640 10 monclient: init 2024-05-15T13:10:24.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:24.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-05-15T13:10:24.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f12640b5410 0x7f12640a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 10 monclient(hunting): picked mon.noname-a con 0x7f12640b5410 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:24.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f126413f990 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:24.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1264146310 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:24.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640ab470 con 0x7f126413f990 2024-05-15T13:10:24.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640aa0c0 con 0x7f1264146310 2024-05-15T13:10:24.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12640b57f0 con 0x7f12640b5410 2024-05-15T13:10:24.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:24.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.015+0000 7f126c71c640 10 monclient(hunting): authenticate will time out at 3782.177222s 2024-05-15T13:10:24.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.016+0000 7f126acb9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1264146310 0x7f126414cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:55258/0 (socket says 172.21.15.49:55258) 2024-05-15T13:10:24.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.016+0000 7f126acb9640 1 -- 172.21.15.49:0/295198651 learned_addr learned my addr 172.21.15.49:0/295198651 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:24.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.029+0000 7f126a4b8640 1 --2- 172.21.15.49:0/295198651 >> v2:172.21.15.49:3300/0 conn(0x7f12640b5410 0x7f12640a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.030+0000 7f126a4b8640 10 monclient(hunting): get_auth_request con 0x7f12640b5410 auth_method 0 2024-05-15T13:10:24.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.030+0000 7f126a4b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.030+0000 7f126a4b8640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.030+0000 7f126a4b8640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.030+0000 7f12694b6640 1 -- 172.21.15.49:0/295198651 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706159996 0 0) 0x7f12640aa0c0 con 0x7f1264146310 2024-05-15T13:10:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.031+0000 7f12694b6640 10 monclient(hunting): _negotiate 2024-05-15T13:10:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.031+0000 7f12694b6640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.031+0000 7f12694b6640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.031+0000 7f12694b6640 10 monclient(hunting): my global_id is 22066 2024-05-15T13:10:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.031+0000 7f12694b6640 1 -- 172.21.15.49:0/295198651 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1250003520 con 0x7f1264146310 2024-05-15T13:10:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.031+0000 7f126a4b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.031+0000 7f126a4b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.031+0000 7f126a4b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.034+0000 7f126a4b8640 10 monclient(hunting): handle_auth_done global_id 16218 payload 274 2024-05-15T13:10:24.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.034+0000 7f126a4b8640 1 -- 172.21.15.49:0/295198651 >> v1:172.21.15.88:6789/0 conn(0x7f1264146310 legacy=0x7f126414cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.034+0000 7f126a4b8640 1 -- 172.21.15.49:0/295198651 >> v1:172.21.15.49:6789/0 conn(0x7f126413f990 legacy=0x7f1264145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:24.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.034+0000 7f126a4b8640 10 monclient: _finish_hunting 0 2024-05-15T13:10:24.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.034+0000 7f126a4b8640 1 monclient: found mon.noname-a 2024-05-15T13:10:24.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.034+0000 7f126a4b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:24.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.034+0000 7f126a4b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T13:10:24.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.034+0000 7f126a4b8640 1 -- 172.21.15.49:0/295198651 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12640aa5e0 con 0x7f12640b5410 2024-05-15T13:10:24.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.036+0000 7f126a4b8640 1 --2- 172.21.15.49:0/295198651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12640b5410 0x7f12640a9960 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7f12600097a0 tx=0x7f126002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2764eeb409868e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:24.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 1 -- 172.21.15.49:0/295198651 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12600315a0 con 0x7f12640b5410 2024-05-15T13:10:24.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:10:24.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 10 monclient: dump: 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 10 monclient: _finish_auth 0 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 10 monclient: _check_auth_tickets 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:24.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 1 -- 172.21.15.49:0/295198651 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1260037070 con 0x7f12640b5410 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 1 -- 172.21.15.49:0/295198651 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260031960 con 0x7f12640b5410 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f126c71c640 5 monclient: authenticate success, global_id 16218 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f126c71c640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f126c71c640 4 set_mon_vals no callback set 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.046+0000 7f12694b6640 10 monclient: dump: 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.038 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.039 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.039 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.039 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.039 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.039 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.047+0000 7f126c71c640 10 monclient: shutdown 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.047+0000 7f126c71c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.047+0000 7f126c71c640 1 -- 172.21.15.49:0/295198651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12640b5410 msgr2=0x7f12640a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.047+0000 7f126c71c640 1 --2- 172.21.15.49:0/295198651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12640b5410 0x7f12640a9960 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7f12600097a0 tx=0x7f126002b840 comp rx=0 tx=0).stop 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.048+0000 7f126c71c640 1 -- 172.21.15.49:0/295198651 shutdown_connections 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.048+0000 7f126c71c640 1 --2- 172.21.15.49:0/295198651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12640b5410 0x7f12640a9960 unknown :-1 s=CLOSED pgs=3269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.048+0000 7f126c71c640 1 -- 172.21.15.49:0/295198651 >> 172.21.15.49:0/295198651 conn(0x7f1264066f00 msgr2=0x7f1264067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.048+0000 7f126c71c640 1 -- 172.21.15.49:0/295198651 shutdown_connections 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.048+0000 7f126c71c640 1 -- 172.21.15.49:0/295198651 wait complete. 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.049+0000 7f126c71c640 10 monclient: build_initial_monmap 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.049+0000 7f126c71c640 10 monclient: monmap: 2024-05-15T13:10:24.042 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.049+0000 7f126c71c640 1 librados: starting msgr at 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.049+0000 7f126c71c640 1 librados: starting objecter 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.049+0000 7f126c71c640 1 Processor -- start 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.049+0000 7f126c71c640 1 -- start start 2024-05-15T13:10:24.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.049+0000 7f126c71c640 1 librados: setting wanted keys 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.049+0000 7f126c71c640 1 librados: calling monclient init 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.049+0000 7f126c71c640 10 monclient: init 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640b5410 0x7f12641455f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 10 monclient(hunting): picked mon.noname-b con 0x7f12640b5410 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f126413f990 0x7f1264143880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 10 monclient(hunting): picked mon.noname-c con 0x7f126413f990 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1264146310 0x7f1264143dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 10 monclient(hunting): picked mon.noname-a con 0x7f1264146310 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126413fe00 con 0x7f1264146310 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12640a9ea0 con 0x7f12640b5410 2024-05-15T13:10:24.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126413ff80 con 0x7f126413f990 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.050+0000 7f126c71c640 10 monclient(hunting): authenticate will time out at 3782.212201s 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.051+0000 7f126a4b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640b5410 0x7f12641455f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.051+0000 7f1269cb7640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f126413f990 0x7f1264143880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.051+0000 7f126a4b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640b5410 0x7f12641455f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:43272/0 (socket says 172.21.15.49:43272) 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.051+0000 7f126a4b8640 1 -- 172.21.15.49:0/320490036 learned_addr learned my addr 172.21.15.49:0/320490036 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.051+0000 7f126a4b8640 10 monclient(hunting): get_auth_request con 0x7f12640b5410 auth_method 0 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.051+0000 7f126a4b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.051+0000 7f126a4b8640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.051+0000 7f126a4b8640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.052+0000 7f1269cb7640 10 monclient(hunting): get_auth_request con 0x7f126413f990 auth_method 0 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.052+0000 7f1269cb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.052+0000 7f1269cb7640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.052+0000 7f1269cb7640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.052+0000 7f126a4b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.052+0000 7f126a4b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.052+0000 7f126a4b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.052+0000 7f1269cb7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.052+0000 7f1269cb7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.052+0000 7f1269cb7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.054+0000 7f126a4b8640 10 monclient(hunting): handle_auth_done global_id 22069 payload 824 2024-05-15T13:10:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.054+0000 7f126a4b8640 1 -- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f126413f990 msgr2=0x7f1264143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.054+0000 7f126a4b8640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f126413f990 0x7f1264143880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.054+0000 7f126a4b8640 1 -- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1264146310 msgr2=0x7f1264143dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.054+0000 7f126a4b8640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1264146310 0x7f1264143dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.054+0000 7f126a4b8640 10 monclient: _finish_hunting 0 2024-05-15T13:10:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.054+0000 7f126a4b8640 1 monclient: found mon.noname-b 2024-05-15T13:10:24.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.054+0000 7f126a4b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:24.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.054+0000 7f126a4b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.054+0000 7f126a4b8640 1 -- 172.21.15.49:0/320490036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1264139270 con 0x7f12640b5410 2024-05-15T13:10:24.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.057+0000 7f126a4b8640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640b5410 0x7f12641455f0 secure :-1 s=READY pgs=5469 cs=0 l=1 rev1=1 crypto rx=0x7f126002be50 tx=0x7f12600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:24.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.057+0000 7f125affd640 10 client.?.objecter ms_handle_connect 0x7f12640b5410 2024-05-15T13:10:24.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.057+0000 7f125affd640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:24.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 1 -- 172.21.15.49:0/320490036 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260032630 con 0x7f12640b5410 2024-05-15T13:10:24.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:24.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 10 monclient: dump: 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 10 monclient: _finish_auth 0 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 10 monclient: _check_auth_tickets 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:24.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 1 -- 172.21.15.49:0/320490036 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12600327f0 con 0x7f12640b5410 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 1 -- 172.21.15.49:0/320490036 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260032be0 con 0x7f12640b5410 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f125affd640 10 monclient: dump: 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.054 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f126c71c640 5 monclient: authenticate success, global_id 22069 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f126c71c640 10 monclient: _renew_subs 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f126c71c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f126c71c640 1 -- 172.21.15.49:0/320490036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1264139500 con 0x7f12640b5410 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f1268cb5640 4 set_mon_vals no callback set 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f126c71c640 10 client.22069.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f126c71c640 10 monclient: _renew_subs 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f126c71c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f126c71c640 1 -- 172.21.15.49:0/320490036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12641399a0 con 0x7f12640b5410 2024-05-15T13:10:24.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.062+0000 7f126c71c640 1 librados: init done 2024-05-15T13:10:24.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.063+0000 7f126c71c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:24.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.063+0000 7f126c71c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:24.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.063+0000 7f126c71c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.063+0000 7f126c71c640 1 -- 172.21.15.49:0/320490036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126404f0e0 con 0x7f12640b5410 2024-05-15T13:10:24.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.359+0000 7f125affd640 1 -- 172.21.15.49:0/320490036 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f126004c020 con 0x7f12640b5410 2024-05-15T13:10:24.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.359+0000 7f125affd640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f124806ce60 0x7f124806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.359+0000 7f1269cb7640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f124806ce60 0x7f124806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.359+0000 7f1269cb7640 10 monclient: get_auth_request con 0x7f124806ce60 auth_method 0 2024-05-15T13:10:24.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.360+0000 7f1269cb7640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f124806ce60 0x7f124806f320 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1264144860 tx=0x7f1254007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:24.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.360+0000 7f125affd640 10 client.22069.objecter ms_handle_connect 0x7f124806ce60 2024-05-15T13:10:24.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.363+0000 7f125affd640 1 -- 172.21.15.49:0/320490036 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f126002f020 con 0x7f12640b5410 2024-05-15T13:10:24.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.363+0000 7f125affd640 10 client.22069.objecter ms_dispatch 0x7f1264000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:24.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.363+0000 7f125affd640 3 client.22069.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:24.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.363+0000 7f125affd640 3 client.22069.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:24.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.363+0000 7f125affd640 20 client.22069.objecter dump_active .. 0 homeless 2024-05-15T13:10:24.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.373+0000 7f125affd640 1 -- 172.21.15.49:0/320490036 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1260040070 con 0x7f12640b5410 2024-05-15T13:10:24.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.373+0000 7f125affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:24.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.373+0000 7f125affd640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:24.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.562+0000 7f126c71c640 1 -- 172.21.15.49:0/320490036 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f1264146ad0 con 0x7f124806ce60 2024-05-15T13:10:24.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.562+0000 7f125affd640 1 -- 172.21.15.49:0/320490036 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1264146ad0 con 0x7f124806ce60 2024-05-15T13:10:24.552 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 5 to deep-scrub 2024-05-15T13:10:24.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 10 librados: watch_flush enter 2024-05-15T13:10:24.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 10 librados: watch_flush exit 2024-05-15T13:10:24.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 20 client.22069.objecter shutdown clearing up homeless session... 2024-05-15T13:10:24.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 10 client.22069.objecter successfully canceled tick 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 -- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f124806ce60 msgr2=0x7f124806f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f124806ce60 0x7f124806f320 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1264144860 tx=0x7f1254007e30 comp rx=0 tx=0).stop 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 10 monclient: shutdown 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 -- 172.21.15.49:0/320490036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640b5410 msgr2=0x7f12641455f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640b5410 0x7f12641455f0 secure :-1 s=READY pgs=5469 cs=0 l=1 rev1=1 crypto rx=0x7f126002be50 tx=0x7f12600046c0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 -- 172.21.15.49:0/320490036 shutdown_connections 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1264146310 0x7f1264143dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f126413f990 0x7f1264143880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f124806ce60 0x7f124806f320 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 --2- 172.21.15.49:0/320490036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640b5410 0x7f12641455f0 unknown :-1 s=CLOSED pgs=5469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 -- 172.21.15.49:0/320490036 >> 172.21.15.49:0/320490036 conn(0x7f1264066f00 msgr2=0x7f126413d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 -- 172.21.15.49:0/320490036 shutdown_connections 2024-05-15T13:10:24.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 -- 172.21.15.49:0/320490036 wait complete. 2024-05-15T13:10:24.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.564+0000 7f126c71c640 1 librados: shutdown 2024-05-15T13:10:24.566 INFO:tasks.ceph:Scrubbing osd.6 2024-05-15T13:10:24.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:10:24.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.683+0000 7f3c481a4640 10 monclient: get_monmap_and_config 2024-05-15T13:10:24.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.683+0000 7f3c481a4640 10 monclient: build_initial_monmap 2024-05-15T13:10:24.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.683+0000 7f3c481a4640 10 monclient: monmap: 2024-05-15T13:10:24.673 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:24.673 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.673 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:24.684114+0000 2024-05-15T13:10:24.674 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:24.684114+0000 2024-05-15T13:10:24.674 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:24.674 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.674 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:24.674 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:24.674 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:24.674 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:24.674 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:24.674 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.684+0000 7f3c481a4640 1 Processor -- start 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.684+0000 7f3c481a4640 1 -- start start 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.684+0000 7f3c481a4640 10 monclient: init 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3c400b5410 0x7f3c400aa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 10 monclient(hunting): picked mon.noname-a con 0x7f3c400b5410 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c400b1f10 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4013f900 0x7f3c4013fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 10 monclient(hunting): picked mon.noname-b con 0x7f3c4013f900 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c400ab5f0 con 0x7f3c400b1f10 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c400b57f0 con 0x7f3c400b5410 2024-05-15T13:10:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c400aa780 con 0x7f3c4013f900 2024-05-15T13:10:24.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:24.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c481a4640 10 monclient(hunting): authenticate will time out at 3782.846673s 2024-05-15T13:10:24.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c4573f640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3c400b1f10 0x7f3c40142770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:38160/0 (socket says 172.21.15.49:38160) 2024-05-15T13:10:24.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c4573f640 1 -- 172.21.15.49:0/1740944642 learned_addr learned my addr 172.21.15.49:0/1740944642 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:24.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.685+0000 7f3c46741640 1 --2- 172.21.15.49:0/1740944642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4013f900 0x7f3c4013fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.686+0000 7f3c46741640 10 monclient(hunting): get_auth_request con 0x7f3c4013f900 auth_method 0 2024-05-15T13:10:24.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.686+0000 7f3c46741640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.686+0000 7f3c46741640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.686+0000 7f3c46741640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.686+0000 7f3c46741640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.686+0000 7f3c46741640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.686+0000 7f3c46741640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c44f3e640 1 -- 172.21.15.49:0/1740944642 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773830659 0 0) 0x7f3c400ab5f0 con 0x7f3c400b1f10 2024-05-15T13:10:24.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c44f3e640 10 monclient(hunting): _negotiate 2024-05-15T13:10:24.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c44f3e640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c44f3e640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c44f3e640 10 monclient(hunting): my global_id is 22072 2024-05-15T13:10:24.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c44f3e640 1 -- 172.21.15.49:0/1740944642 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c2c003520 con 0x7f3c400b1f10 2024-05-15T13:10:24.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c46741640 10 monclient(hunting): handle_auth_done global_id 17648 payload 274 2024-05-15T13:10:24.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c46741640 1 -- 172.21.15.49:0/1740944642 >> v2:172.21.15.49:3300/0 conn(0x7f3c400b5410 msgr2=0x7f3c400aa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:24.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c46741640 1 --2- 172.21.15.49:0/1740944642 >> v2:172.21.15.49:3300/0 conn(0x7f3c400b5410 0x7f3c400aa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c46741640 1 -- 172.21.15.49:0/1740944642 >> v1:172.21.15.88:6789/0 conn(0x7f3c400b1f10 legacy=0x7f3c40142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c46741640 10 monclient: _finish_hunting 0 2024-05-15T13:10:24.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c46741640 1 monclient: found mon.noname-b 2024-05-15T13:10:24.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c46741640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:24.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c46741640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T13:10:24.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.688+0000 7f3c46741640 1 -- 172.21.15.49:0/1740944642 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c400b4a30 con 0x7f3c4013f900 2024-05-15T13:10:24.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.690+0000 7f3c46741640 1 --2- 172.21.15.49:0/1740944642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4013f900 0x7f3c4013fd00 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009d90 tx=0x7f3c3802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4204cc888a0e4346 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:24.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 1 -- 172.21.15.49:0/1740944642 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38039070 con 0x7f3c4013f900 2024-05-15T13:10:24.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:24.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 10 monclient: dump: 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 10 monclient: _finish_auth 0 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 10 monclient: _check_auth_tickets 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:24.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 1 -- 172.21.15.49:0/1740944642 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3802cc50 con 0x7f3c4013f900 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c481a4640 5 monclient: authenticate success, global_id 17648 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c481a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c481a4640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 1 -- 172.21.15.49:0/1740944642 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38034440 con 0x7f3c4013f900 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c481a4640 4 set_mon_vals no callback set 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.694+0000 7f3c44f3e640 10 monclient: dump: 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.686 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.687 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.687 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.687 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.687 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.695+0000 7f3c481a4640 10 monclient: shutdown 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.695+0000 7f3c481a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.695+0000 7f3c481a4640 1 -- 172.21.15.49:0/1740944642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4013f900 msgr2=0x7f3c4013fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.695+0000 7f3c481a4640 1 --2- 172.21.15.49:0/1740944642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4013f900 0x7f3c4013fd00 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009d90 tx=0x7f3c3802bea0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.695+0000 7f3c481a4640 1 -- 172.21.15.49:0/1740944642 shutdown_connections 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.695+0000 7f3c481a4640 1 --2- 172.21.15.49:0/1740944642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4013f900 0x7f3c4013fd00 unknown :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.695+0000 7f3c481a4640 1 --2- 172.21.15.49:0/1740944642 >> v2:172.21.15.49:3300/0 conn(0x7f3c400b5410 0x7f3c400aa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.695+0000 7f3c481a4640 1 -- 172.21.15.49:0/1740944642 >> 172.21.15.49:0/1740944642 conn(0x7f3c40066f30 msgr2=0x7f3c40067340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.695+0000 7f3c481a4640 1 -- 172.21.15.49:0/1740944642 shutdown_connections 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.695+0000 7f3c481a4640 1 -- 172.21.15.49:0/1740944642 wait complete. 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.696+0000 7f3c481a4640 10 monclient: build_initial_monmap 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.696+0000 7f3c481a4640 10 monclient: monmap: 2024-05-15T13:10:24.690 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.696+0000 7f3c481a4640 1 librados: starting msgr at 2024-05-15T13:10:24.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.696+0000 7f3c481a4640 1 librados: starting objecter 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 1 Processor -- start 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 1 -- start start 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 1 librados: setting wanted keys 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 1 librados: calling monclient init 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient: init 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c400b1f10 0x7f3c4013ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient(hunting): picked mon.noname-c con 0x7f3c400b1f10 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c400b5410 0x7f3c4013b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient(hunting): picked mon.noname-b con 0x7f3c400b5410 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c4013f900 0x7f3c40138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient(hunting): picked mon.noname-a con 0x7f3c4013f900 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c40140240 con 0x7f3c4013f900 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c400a9de0 con 0x7f3c400b5410 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c401403c0 con 0x7f3c400b1f10 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.697+0000 7f3c481a4640 10 monclient(hunting): authenticate will time out at 3782.859115s 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c4573f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c400b5410 0x7f3c4013b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c45f40640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c400b1f10 0x7f3c4013ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c4573f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c400b5410 0x7f3c4013b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35910/0 (socket says 172.21.15.49:35910) 2024-05-15T13:10:24.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c4573f640 1 -- 172.21.15.49:0/1783416957 learned_addr learned my addr 172.21.15.49:0/1783416957 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c4573f640 10 monclient(hunting): get_auth_request con 0x7f3c400b5410 auth_method 0 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c4573f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c4573f640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c4573f640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c45f40640 10 monclient(hunting): get_auth_request con 0x7f3c400b1f10 auth_method 0 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c45f40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c45f40640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c45f40640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.698+0000 7f3c4573f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.699+0000 7f3c4573f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.699+0000 7f3c4573f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.699+0000 7f3c45f40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.699+0000 7f3c45f40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.699+0000 7f3c45f40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.701+0000 7f3c4573f640 10 monclient(hunting): handle_auth_done global_id 22075 payload 824 2024-05-15T13:10:24.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.701+0000 7f3c4573f640 1 -- 172.21.15.49:0/1783416957 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c400b1f10 msgr2=0x7f3c4013ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.701+0000 7f3c4573f640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c400b1f10 0x7f3c4013ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.701+0000 7f3c4573f640 1 -- 172.21.15.49:0/1783416957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c4013f900 msgr2=0x7f3c40138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.701+0000 7f3c4573f640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c4013f900 0x7f3c40138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.701+0000 7f3c4573f640 10 monclient: _finish_hunting 0 2024-05-15T13:10:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.701+0000 7f3c4573f640 1 monclient: found mon.noname-b 2024-05-15T13:10:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.701+0000 7f3c4573f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.701+0000 7f3c4573f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.701+0000 7f3c4573f640 1 -- 172.21.15.49:0/1783416957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c400ac300 con 0x7f3c400b5410 2024-05-15T13:10:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.704+0000 7f3c4573f640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c400b5410 0x7f3c4013b650 secure :-1 s=READY pgs=5471 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000c500 tx=0x7f3c3000c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.704+0000 7f3c367fc640 10 client.?.objecter ms_handle_connect 0x7f3c400b5410 2024-05-15T13:10:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.704+0000 7f3c367fc640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:24.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 1 -- 172.21.15.49:0/1783416957 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30017070 con 0x7f3c400b5410 2024-05-15T13:10:24.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 10 monclient: dump: 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 10 monclient: _finish_auth 0 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 10 monclient: _check_auth_tickets 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:24.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 1 -- 172.21.15.49:0/1783416957 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c30005ce0 con 0x7f3c400b5410 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 1 -- 172.21.15.49:0/1783416957 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c300072b0 con 0x7f3c400b5410 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c367fc640 10 monclient: dump: 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c481a4640 5 monclient: authenticate success, global_id 22075 2024-05-15T13:10:24.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c481a4640 10 monclient: _renew_subs 2024-05-15T13:10:24.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c481a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c400ac570 con 0x7f3c400b5410 2024-05-15T13:10:24.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c37fff640 4 set_mon_vals no callback set 2024-05-15T13:10:24.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c481a4640 10 client.22075.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:24.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c481a4640 10 monclient: _renew_subs 2024-05-15T13:10:24.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c481a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4013bf90 con 0x7f3c400b5410 2024-05-15T13:10:24.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.708+0000 7f3c481a4640 1 librados: init done 2024-05-15T13:10:24.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.709+0000 7f3c481a4640 10 client.22075.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:24.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.709+0000 7f3c481a4640 15 client.22075.objecter _session_command_op_assign -1 1 2024-05-15T13:10:24.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.709+0000 7f3c481a4640 20 client.22075.objecter _get_session osd=-1 returning homeless 2024-05-15T13:10:24.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.709+0000 7f3c481a4640 10 client.22075.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:24.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.709+0000 7f3c481a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.709+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3c4004f0e0 con 0x7f3c400b5410 2024-05-15T13:10:24.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.728+0000 7f3c367fc640 1 -- 172.21.15.49:0/1783416957 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3c3001a070 con 0x7f3c400b5410 2024-05-15T13:10:24.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.729+0000 7f3c367fc640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f3c2406ceb0 0x7f3c2406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.729+0000 7f3c45f40640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f3c2406ceb0 0x7f3c2406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.729+0000 7f3c45f40640 10 monclient: get_auth_request con 0x7f3c2406ceb0 auth_method 0 2024-05-15T13:10:24.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.729+0000 7f3c45f40640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f3c2406ceb0 0x7f3c2406f370 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009990 tx=0x7f3c3c009110 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:24.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.729+0000 7f3c367fc640 10 client.22075.objecter ms_handle_connect 0x7f3c2406ceb0 2024-05-15T13:10:24.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 1 -- 172.21.15.49:0/1783416957 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f3c30056310 con 0x7f3c400b5410 2024-05-15T13:10:24.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 10 client.22075.objecter ms_dispatch 0x7f3c40000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:24.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 3 client.22075.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:24.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 3 client.22075.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:24.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 10 client.22075.objecter checking command 1 2024-05-15T13:10:24.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] conn(0x7f3c240758a0 0x7f3c24077d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 20 client.22075.objecter _get_session s=0x7f3c240751d0 osd=6 3 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 20 client.22075.objecter put_session s=0x7f3c240751d0 osd=6 3 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 15 client.22075.objecter _session_command_op_remove -1 1 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 20 client.22075.objecter _get_session s=0x7f3c240751d0 osd=6 3 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 20 client.22075.objecter get_session s=0x7f3c240751d0 osd=6 3 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 15 client.22075.objecter _session_command_op_assign 6 1 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 20 client.22075.objecter put_session s=0x7f3c240751d0 osd=6 4 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 10 client.22075.objecter _send_command 1 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 1 -- 172.21.15.49:0/1783416957 --> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c3001ee50 con 0x7f3c240758a0 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 20 client.22075.objecter dump_active .. 0 homeless 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 1 -- 172.21.15.49:0/1783416957 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=1251) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c30054510 con 0x7f3c400b5410 2024-05-15T13:10:24.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.731+0000 7f3c367fc640 10 monclient: handle_get_version_reply finishing 1 version 1251 2024-05-15T13:10:24.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.732+0000 7f3c46741640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] conn(0x7f3c240758a0 0x7f3c24077d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.732+0000 7f3c46741640 10 monclient: get_auth_request con 0x7f3c240758a0 auth_method 0 2024-05-15T13:10:24.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.736+0000 7f3c46741640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] conn(0x7f3c240758a0 0x7f3c24077d40 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:24.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.736+0000 7f3c367fc640 10 client.22075.objecter ms_handle_connect 0x7f3c240758a0 2024-05-15T13:10:24.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.759+0000 7f3c367fc640 1 -- 172.21.15.49:0/1783416957 <== osd.6 v2:172.21.15.88:6800/3138490191 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c3001ee50 con 0x7f3c240758a0 2024-05-15T13:10:24.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.759+0000 7f3c367fc640 10 client.22075.objecter ms_dispatch 0x7f3c40000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:10:24.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.759+0000 7f3c367fc640 10 client.22075.objecter _finish_command 1 = system:0 2024-05-15T13:10:24.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.759+0000 7f3c367fc640 20 client.22075.objecter put_session s=0x7f3c240751d0 osd=6 4 2024-05-15T13:10:24.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.759+0000 7f3c367fc640 15 client.22075.objecter _session_command_op_remove 6 1 2024-05-15T13:10:24.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.772+0000 7f3c481a4640 10 client.22075.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:10:24.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.772+0000 7f3c481a4640 15 client.22075.objecter _session_command_op_assign -1 2 2024-05-15T13:10:24.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.773+0000 7f3c481a4640 20 client.22075.objecter _get_session s=0x7f3c240751d0 osd=6 3 2024-05-15T13:10:24.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.773+0000 7f3c481a4640 20 client.22075.objecter put_session s=0x7f3c240751d0 osd=6 3 2024-05-15T13:10:24.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.773+0000 7f3c481a4640 20 client.22075.objecter _get_session s=0x7f3c240751d0 osd=6 3 2024-05-15T13:10:24.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.773+0000 7f3c481a4640 15 client.22075.objecter _session_command_op_remove -1 2 2024-05-15T13:10:24.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.773+0000 7f3c481a4640 20 client.22075.objecter get_session s=0x7f3c240751d0 osd=6 3 2024-05-15T13:10:24.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.773+0000 7f3c481a4640 15 client.22075.objecter _session_command_op_assign 6 2 2024-05-15T13:10:24.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.773+0000 7f3c481a4640 20 client.22075.objecter put_session s=0x7f3c240751d0 osd=6 4 2024-05-15T13:10:24.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.773+0000 7f3c481a4640 10 client.22075.objecter _send_command 2 2024-05-15T13:10:24.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.773+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 --> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3c400ab2a0 con 0x7f3c240758a0 2024-05-15T13:10:24.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c367fc640 1 -- 172.21.15.49:0/1783416957 <== osd.6 v2:172.21.15.88:6800/3138490191 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3c400ab2a0 con 0x7f3c240758a0 2024-05-15T13:10:24.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c367fc640 10 client.22075.objecter ms_dispatch 0x7f3c40000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:10:24.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c367fc640 10 client.22075.objecter _finish_command 2 = system:0 2024-05-15T13:10:24.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c367fc640 20 client.22075.objecter put_session s=0x7f3c240751d0 osd=6 4 2024-05-15T13:10:24.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c367fc640 15 client.22075.objecter _session_command_op_remove 6 2 2024-05-15T13:10:24.776 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-05-15T13:10:24.776 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:10:24.776 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-05-15T13:10:24.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 10 librados: watch_flush enter 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 10 librados: watch_flush exit 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 10 client.22075.objecter close_session for osd.6 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 >> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] conn(0x7f3c240758a0 msgr2=0x7f3c24077d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.88:6800/3138490191,v1:172.21.15.88:6801/3138490191] conn(0x7f3c240758a0 0x7f3c24077d40 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 20 client.22075.objecter put_session s=0x7f3c240751d0 osd=6 1 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 20 client.22075.objecter shutdown clearing up homeless session... 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 10 client.22075.objecter successfully canceled tick 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f3c2406ceb0 msgr2=0x7f3c2406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f3c2406ceb0 0x7f3c2406f370 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009990 tx=0x7f3c3c009110 comp rx=0 tx=0).stop 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 10 monclient: shutdown 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c400b5410 msgr2=0x7f3c4013b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.786+0000 7f3c481a4640 1 --2- 172.21.15.49:0/1783416957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c400b5410 0x7f3c4013b650 secure :-1 s=READY pgs=5471 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000c500 tx=0x7f3c3000c900 comp rx=0 tx=0).stop 2024-05-15T13:10:24.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.787+0000 7f3c46741640 1 -- 172.21.15.49:0/1783416957 reap_dead start 2024-05-15T13:10:24.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.787+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 shutdown_connections 2024-05-15T13:10:24.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.787+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 >> 172.21.15.49:0/1783416957 conn(0x7f3c40066f30 msgr2=0x7f3c400b2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:24.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.787+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 shutdown_connections 2024-05-15T13:10:24.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.787+0000 7f3c481a4640 1 -- 172.21.15.49:0/1783416957 wait complete. 2024-05-15T13:10:24.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.787+0000 7f3c481a4640 1 librados: shutdown 2024-05-15T13:10:24.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.901+0000 7fc8798cf640 10 monclient: get_monmap_and_config 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.901+0000 7fc8798cf640 10 monclient: build_initial_monmap 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.901+0000 7fc8798cf640 10 monclient: monmap: 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:24.901910+0000 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:24.901910+0000 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:24.892 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:24.892 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:24.892 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 1 Processor -- start 2024-05-15T13:10:24.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 1 -- start start 2024-05-15T13:10:24.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient: init 2024-05-15T13:10:24.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8740b5470 0x7fc8740a9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient(hunting): picked mon.noname-c con 0x7fc8740b5470 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc874142660 0x7fc874142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient(hunting): picked mon.noname-b con 0x7fc874142660 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc87413f8b0 0x7fc87413fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient(hunting): picked mon.noname-a con 0x7fc87413f8b0 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8740b5850 con 0x7fc87413f8b0 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8740a9d90 con 0x7fc8740b5470 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc874143000 con 0x7fc874142660 2024-05-15T13:10:24.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.902+0000 7fc8798cf640 10 monclient(hunting): authenticate will time out at 3783.064159s 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc872ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8740b5470 0x7fc8740a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc8737fe640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc87413f8b0 0x7fc87413fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc8727fc640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc874142660 0x7fc874142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc872ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8740b5470 0x7fc8740a9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35914/0 (socket says 172.21.15.49:35914) 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc872ffd640 1 -- 172.21.15.49:0/3916574885 learned_addr learned my addr 172.21.15.49:0/3916574885 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc872ffd640 10 monclient(hunting): get_auth_request con 0x7fc8740b5470 auth_method 0 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc872ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc872ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc872ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc8737fe640 10 monclient(hunting): get_auth_request con 0x7fc87413f8b0 auth_method 0 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc8737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc8737fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc8737fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.903+0000 7fc8727fc640 10 monclient(hunting): get_auth_request con 0x7fc874142660 auth_method 0 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc8727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc8727fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc8727fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc872ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc872ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc872ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc8737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc8737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc8737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc8727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc8727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.904+0000 7fc8727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.905+0000 7fc872ffd640 10 monclient(hunting): handle_auth_done global_id 22078 payload 274 2024-05-15T13:10:24.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.905+0000 7fc872ffd640 1 -- 172.21.15.49:0/3916574885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc874142660 msgr2=0x7fc874142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:24.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.905+0000 7fc872ffd640 1 --2- 172.21.15.49:0/3916574885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc874142660 0x7fc874142ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.905+0000 7fc872ffd640 1 -- 172.21.15.49:0/3916574885 >> v2:172.21.15.49:3300/0 conn(0x7fc87413f8b0 msgr2=0x7fc87413fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:24.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.905+0000 7fc872ffd640 1 --2- 172.21.15.49:0/3916574885 >> v2:172.21.15.49:3300/0 conn(0x7fc87413f8b0 0x7fc87413fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.905+0000 7fc872ffd640 10 monclient: _finish_hunting 0 2024-05-15T13:10:24.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.905+0000 7fc872ffd640 1 monclient: found mon.noname-c 2024-05-15T13:10:24.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.905+0000 7fc872ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:24.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.905+0000 7fc872ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.905+0000 7fc872ffd640 1 -- 172.21.15.49:0/3916574885 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc874140260 con 0x7fc8740b5470 2024-05-15T13:10:24.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.907+0000 7fc872ffd640 1 --2- 172.21.15.49:0/3916574885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8740b5470 0x7fc8740a9850 secure :-1 s=READY pgs=5472 cs=0 l=1 rev1=1 crypto rx=0x7fc8740b4c10 tx=0x7fc86402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7762529dd2efbc server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 1 -- 172.21.15.49:0/3916574885 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864035020 con 0x7fc8740b5470 2024-05-15T13:10:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 10 monclient: dump: 2024-05-15T13:10:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.901 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 10 monclient: _finish_auth 0 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 10 monclient: _check_auth_tickets 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 1 -- 172.21.15.49:0/3916574885 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc864031870 con 0x7fc8740b5470 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 1 -- 172.21.15.49:0/3916574885 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864031c40 con 0x7fc8740b5470 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc8798cf640 5 monclient: authenticate success, global_id 22078 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc8798cf640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc8798cf640 4 set_mon_vals no callback set 2024-05-15T13:10:24.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.911+0000 7fc871ffb640 10 monclient: dump: 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.903 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 10 monclient: shutdown 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 1 -- 172.21.15.49:0/3916574885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8740b5470 msgr2=0x7fc8740a9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 1 --2- 172.21.15.49:0/3916574885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8740b5470 0x7fc8740a9850 secure :-1 s=READY pgs=5472 cs=0 l=1 rev1=1 crypto rx=0x7fc8740b4c10 tx=0x7fc86402b910 comp rx=0 tx=0).stop 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 1 -- 172.21.15.49:0/3916574885 shutdown_connections 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 1 --2- 172.21.15.49:0/3916574885 >> v2:172.21.15.49:3300/0 conn(0x7fc87413f8b0 0x7fc87413fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 1 --2- 172.21.15.49:0/3916574885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc874142660 0x7fc874142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 1 --2- 172.21.15.49:0/3916574885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8740b5470 0x7fc8740a9850 unknown :-1 s=CLOSED pgs=5472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 1 -- 172.21.15.49:0/3916574885 >> 172.21.15.49:0/3916574885 conn(0x7fc874066f00 msgr2=0x7fc874067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 1 -- 172.21.15.49:0/3916574885 shutdown_connections 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.912+0000 7fc8798cf640 1 -- 172.21.15.49:0/3916574885 wait complete. 2024-05-15T13:10:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.913+0000 7fc8798cf640 10 monclient: build_initial_monmap 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.913+0000 7fc8798cf640 10 monclient: monmap: 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.913+0000 7fc8798cf640 1 librados: starting msgr at 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.913+0000 7fc8798cf640 1 librados: starting objecter 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.913+0000 7fc8798cf640 1 Processor -- start 2024-05-15T13:10:24.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.913+0000 7fc8798cf640 1 -- start start 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.913+0000 7fc8798cf640 1 librados: setting wanted keys 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.913+0000 7fc8798cf640 1 librados: calling monclient init 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.913+0000 7fc8798cf640 10 monclient: init 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8740b5470 0x7fc8741cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 10 monclient(hunting): picked mon.noname-c con 0x7fc8740b5470 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc87413f8b0 0x7fc8741cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 10 monclient(hunting): picked mon.noname-b con 0x7fc87413f8b0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc874142660 0x7fc8741ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 10 monclient(hunting): picked mon.noname-a con 0x7fc874142660 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740aa4d0 con 0x7fc874142660 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740a9fb0 con 0x7fc87413f8b0 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8741ce9c0 con 0x7fc8740b5470 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8798cf640 10 monclient(hunting): authenticate will time out at 3783.075939s 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8737fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc874142660 0x7fc8741ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.914+0000 7fc8727fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc87413f8b0 0x7fc8741cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc872ffd640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8740b5470 0x7fc8741cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8727fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc87413f8b0 0x7fc8741cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35930/0 (socket says 172.21.15.49:35930) 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8727fc640 1 -- 172.21.15.49:0/794976043 learned_addr learned my addr 172.21.15.49:0/794976043 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8727fc640 10 monclient(hunting): get_auth_request con 0x7fc87413f8b0 auth_method 0 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8727fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8727fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8737fe640 10 monclient(hunting): get_auth_request con 0x7fc874142660 auth_method 0 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8737fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8737fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc872ffd640 10 monclient(hunting): get_auth_request con 0x7fc8740b5470 auth_method 0 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc872ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc872ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc872ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc8737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc872ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:24.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc872ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:24.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.915+0000 7fc872ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.918+0000 7fc8727fc640 10 monclient(hunting): handle_auth_done global_id 22081 payload 824 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.918+0000 7fc8727fc640 1 -- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8740b5470 msgr2=0x7fc8741cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.918+0000 7fc8727fc640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8740b5470 0x7fc8741cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.918+0000 7fc8727fc640 1 -- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc874142660 msgr2=0x7fc8741ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.918+0000 7fc8727fc640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc874142660 0x7fc8741ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.918+0000 7fc8727fc640 10 monclient: _finish_hunting 0 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.918+0000 7fc8727fc640 1 monclient: found mon.noname-b 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.918+0000 7fc8727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.918+0000 7fc8727fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.918+0000 7fc8727fc640 1 -- 172.21.15.49:0/794976043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8741cf610 con 0x7fc87413f8b0 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.920+0000 7fc8727fc640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc87413f8b0 0x7fc8741cc800 secure :-1 s=READY pgs=5473 cs=0 l=1 rev1=1 crypto rx=0x7fc86800e870 tx=0x7fc86800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.920+0000 7fc85bfff640 10 client.?.objecter ms_handle_connect 0x7fc87413f8b0 2024-05-15T13:10:24.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.920+0000 7fc85bfff640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:24.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc85bfff640 1 -- 172.21.15.49:0/794976043 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc868017070 con 0x7fc87413f8b0 2024-05-15T13:10:24.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc85bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc85bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:24.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc85bfff640 10 monclient: dump: 2024-05-15T13:10:24.915 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.915 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.915 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.915 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.915 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc85bfff640 10 monclient: _finish_auth 0 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc85bfff640 10 monclient: _check_auth_tickets 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc85bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc85bfff640 1 -- 172.21.15.49:0/794976043 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86800c940 con 0x7fc87413f8b0 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc85bfff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc85bfff640 1 -- 172.21.15.49:0/794976043 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86800cd30 con 0x7fc87413f8b0 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc8798cf640 5 monclient: authenticate success, global_id 22081 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc8798cf640 10 monclient: _renew_subs 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc8798cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.923+0000 7fc8798cf640 1 -- 172.21.15.49:0/794976043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8741cd280 con 0x7fc87413f8b0 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.924+0000 7fc85bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.924+0000 7fc85bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.924+0000 7fc85bfff640 10 monclient: dump: 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.924+0000 7fc8717fa640 4 set_mon_vals no callback set 2024-05-15T13:10:24.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.924+0000 7fc8798cf640 10 client.22081.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:24.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.924+0000 7fc8798cf640 10 monclient: _renew_subs 2024-05-15T13:10:24.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.924+0000 7fc8798cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.924+0000 7fc8798cf640 1 -- 172.21.15.49:0/794976043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc874139220 con 0x7fc87413f8b0 2024-05-15T13:10:24.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.924+0000 7fc8798cf640 1 librados: init done 2024-05-15T13:10:24.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.925+0000 7fc8798cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:24.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.925+0000 7fc8798cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:24.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.925+0000 7fc8798cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:24.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:24.925+0000 7fc8798cf640 1 -- 172.21.15.49:0/794976043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87413cd90 con 0x7fc87413f8b0 2024-05-15T13:10:25.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.177+0000 7fc85bfff640 1 -- 172.21.15.49:0/794976043 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc86801a070 con 0x7fc87413f8b0 2024-05-15T13:10:25.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.178+0000 7fc85bfff640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc86006ce60 0x7fc86006f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.178+0000 7fc872ffd640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc86006ce60 0x7fc86006f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.178+0000 7fc872ffd640 10 monclient: get_auth_request con 0x7fc86006ce60 auth_method 0 2024-05-15T13:10:25.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.178+0000 7fc85bfff640 10 client.22081.objecter ms_handle_connect 0x7fc86006ce60 2024-05-15T13:10:25.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.178+0000 7fc872ffd640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc86006ce60 0x7fc86006f320 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc8740b4c10 tx=0x7fc86403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:25.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.181+0000 7fc85bfff640 1 -- 172.21.15.49:0/794976043 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fc868057050 con 0x7fc87413f8b0 2024-05-15T13:10:25.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.181+0000 7fc85bfff640 10 client.22081.objecter ms_dispatch 0x7fc874000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:25.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.181+0000 7fc85bfff640 3 client.22081.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:25.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.181+0000 7fc85bfff640 3 client.22081.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:25.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.181+0000 7fc85bfff640 20 client.22081.objecter dump_active .. 0 homeless 2024-05-15T13:10:25.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.192+0000 7fc85bfff640 1 -- 172.21.15.49:0/794976043 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc87413cd90 con 0x7fc87413f8b0 2024-05-15T13:10:25.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.192+0000 7fc85bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:25.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.192+0000 7fc85bfff640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:25.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.379+0000 7fc8798cf640 1 -- 172.21.15.49:0/794976043 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fc87404f0e0 con 0x7fc86006ce60 2024-05-15T13:10:25.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.379+0000 7fc85bfff640 1 -- 172.21.15.49:0/794976043 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc87404f0e0 con 0x7fc86006ce60 2024-05-15T13:10:25.369 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 6 to deep-scrub 2024-05-15T13:10:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 10 librados: watch_flush enter 2024-05-15T13:10:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 10 librados: watch_flush exit 2024-05-15T13:10:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 20 client.22081.objecter shutdown clearing up homeless session... 2024-05-15T13:10:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 10 client.22081.objecter successfully canceled tick 2024-05-15T13:10:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 -- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc86006ce60 msgr2=0x7fc86006f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc86006ce60 0x7fc86006f320 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc8740b4c10 tx=0x7fc86403c040 comp rx=0 tx=0).stop 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 10 monclient: shutdown 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 -- 172.21.15.49:0/794976043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc87413f8b0 msgr2=0x7fc8741cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc87413f8b0 0x7fc8741cc800 secure :-1 s=READY pgs=5473 cs=0 l=1 rev1=1 crypto rx=0x7fc86800e870 tx=0x7fc86800ed30 comp rx=0 tx=0).stop 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 -- 172.21.15.49:0/794976043 shutdown_connections 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc874142660 0x7fc8741ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7fc86006ce60 0x7fc86006f320 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc87413f8b0 0x7fc8741cc800 unknown :-1 s=CLOSED pgs=5473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 --2- 172.21.15.49:0/794976043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8740b5470 0x7fc8741cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 -- 172.21.15.49:0/794976043 >> 172.21.15.49:0/794976043 conn(0x7fc874066f00 msgr2=0x7fc87413a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 -- 172.21.15.49:0/794976043 shutdown_connections 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 -- 172.21.15.49:0/794976043 wait complete. 2024-05-15T13:10:25.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.381+0000 7fc8798cf640 1 librados: shutdown 2024-05-15T13:10:25.387 INFO:tasks.ceph:Scrubbing osd.7 2024-05-15T13:10:25.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.506+0000 7f2ded5d1640 10 monclient: get_monmap_and_config 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.506+0000 7f2ded5d1640 10 monclient: build_initial_monmap 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.506+0000 7f2ded5d1640 10 monclient: monmap: 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:25.506828+0000 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:25.506828+0000 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:25.496 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:25.497 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:25.497 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:25.497 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:25.497 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.506+0000 7f2ded5d1640 1 Processor -- start 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 1 -- start start 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 10 monclient: init 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2de80b8a90 0x7f2de80aa410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 10 monclient(hunting): picked mon.noname-a con 0x7f2de80b8a90 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 0x7f2de8145d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 10 monclient(hunting): picked mon.noname-b con 0x7f2de80a99d0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2de80b1ef0 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de80ab5f0 con 0x7f2de80b1ef0 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de80b8e70 con 0x7f2de80b8a90 2024-05-15T13:10:25.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80a9ef0 con 0x7f2de80a99d0 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.507+0000 7f2ded5d1640 10 monclient(hunting): authenticate will time out at 3783.669169s 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2de6ffd640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2de80b8a90 0x7f2de80aa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2dde7fc640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 0x7f2de8145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2de77fe640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f2de80b1ef0 0x7f2de813fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:56274/0 (socket says 172.21.15.49:56274) 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2de77fe640 1 -- 172.21.15.49:0/230802741 learned_addr learned my addr 172.21.15.49:0/230802741 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2dde7fc640 10 monclient(hunting): get_auth_request con 0x7f2de80a99d0 auth_method 0 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2dde7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2dde7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2dde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2de6ffd640 10 monclient(hunting): get_auth_request con 0x7f2de80b8a90 auth_method 0 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2de6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:25.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2de6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:25.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.508+0000 7f2de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:25.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.509+0000 7f2dde7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:25.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.509+0000 7f2dde7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:25.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.509+0000 7f2dde7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.509+0000 7f2de6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.509+0000 7f2de6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.509+0000 7f2de6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.511+0000 7f2dde7fc640 10 monclient(hunting): handle_auth_done global_id 17660 payload 274 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.511+0000 7f2dde7fc640 1 -- 172.21.15.49:0/230802741 >> v2:172.21.15.49:3300/0 conn(0x7f2de80b8a90 msgr2=0x7f2de80aa410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.511+0000 7f2dde7fc640 1 --2- 172.21.15.49:0/230802741 >> v2:172.21.15.49:3300/0 conn(0x7f2de80b8a90 0x7f2de80aa410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.511+0000 7f2dde7fc640 1 -- 172.21.15.49:0/230802741 >> v1:172.21.15.49:6789/0 conn(0x7f2de80b1ef0 legacy=0x7f2de813fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.511+0000 7f2dde7fc640 10 monclient: _finish_hunting 0 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.511+0000 7f2dde7fc640 1 monclient: found mon.noname-b 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.511+0000 7f2dde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.511+0000 7f2dde7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T13:10:25.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.511+0000 7f2dde7fc640 1 -- 172.21.15.49:0/230802741 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de81402c0 con 0x7f2de80a99d0 2024-05-15T13:10:25.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.513+0000 7f2dde7fc640 1 --2- 172.21.15.49:0/230802741 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 0x7f2de8145d00 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 crypto rx=0x7f2dd00097a0 tx=0x7f2dd002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e92b26177afaca9a server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:25.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 1 -- 172.21.15.49:0/230802741 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0035020 con 0x7f2de80a99d0 2024-05-15T13:10:25.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:25.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:25.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 10 monclient: dump: 2024-05-15T13:10:25.507 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:25.507 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 10 monclient: _finish_auth 0 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 10 monclient: _check_auth_tickets 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 1 -- 172.21.15.49:0/230802741 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd0031830 con 0x7f2de80a99d0 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:25.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 1 -- 172.21.15.49:0/230802741 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0031c00 con 0x7f2de80a99d0 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2ded5d1640 5 monclient: authenticate success, global_id 17660 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2ded5d1640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2ded5d1640 4 set_mon_vals no callback set 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.517+0000 7f2de67fc640 10 monclient: dump: 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:25.510 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.518+0000 7f2ded5d1640 10 monclient: shutdown 2024-05-15T13:10:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.518+0000 7f2ded5d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.518+0000 7f2ded5d1640 1 -- 172.21.15.49:0/230802741 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 msgr2=0x7f2de8145d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.518+0000 7f2ded5d1640 1 --2- 172.21.15.49:0/230802741 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 0x7f2de8145d00 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 crypto rx=0x7f2dd00097a0 tx=0x7f2dd002b8d0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.518+0000 7f2ded5d1640 1 -- 172.21.15.49:0/230802741 shutdown_connections 2024-05-15T13:10:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.518+0000 7f2ded5d1640 1 --2- 172.21.15.49:0/230802741 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 0x7f2de8145d00 unknown :-1 s=CLOSED pgs=3124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.518+0000 7f2ded5d1640 1 --2- 172.21.15.49:0/230802741 >> v2:172.21.15.49:3300/0 conn(0x7f2de80b8a90 0x7f2de80aa410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.518+0000 7f2ded5d1640 1 -- 172.21.15.49:0/230802741 >> 172.21.15.49:0/230802741 conn(0x7f2de8067110 msgr2=0x7f2de80b8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.518+0000 7f2ded5d1640 1 -- 172.21.15.49:0/230802741 shutdown_connections 2024-05-15T13:10:25.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.518+0000 7f2ded5d1640 1 -- 172.21.15.49:0/230802741 wait complete. 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.519+0000 7f2ded5d1640 10 monclient: build_initial_monmap 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.519+0000 7f2ded5d1640 10 monclient: monmap: 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.519+0000 7f2ded5d1640 1 librados: starting msgr at 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.519+0000 7f2ded5d1640 1 librados: starting objecter 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.519+0000 7f2ded5d1640 1 Processor -- start 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.519+0000 7f2ded5d1640 1 -- start start 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.519+0000 7f2ded5d1640 1 librados: setting wanted keys 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.519+0000 7f2ded5d1640 1 librados: calling monclient init 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.519+0000 7f2ded5d1640 10 monclient: init 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 0x7f2de813aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 10 monclient(hunting): picked mon.noname-c con 0x7f2de80a99d0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de80b1ef0 0x7f2de813b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 10 monclient(hunting): picked mon.noname-a con 0x7f2de80b1ef0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de80b8a90 0x7f2de81d8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 10 monclient(hunting): picked mon.noname-b con 0x7f2de80b8a90 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de814d090 con 0x7f2de80b1ef0 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de8146450 con 0x7f2de80b8a90 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de814d210 con 0x7f2de80a99d0 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.520+0000 7f2ded5d1640 10 monclient(hunting): authenticate will time out at 3783.682010s 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2dde7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de80b1ef0 0x7f2de813b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2de6ffd640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 0x7f2de813aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2de77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de80b8a90 0x7f2de81d8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2dde7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de80b1ef0 0x7f2de813b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60882/0 (socket says 172.21.15.49:60882) 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2dde7fc640 1 -- 172.21.15.49:0/1977292532 learned_addr learned my addr 172.21.15.49:0/1977292532 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2dde7fc640 10 monclient(hunting): get_auth_request con 0x7f2de80b1ef0 auth_method 0 2024-05-15T13:10:25.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2dde7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2dde7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2dde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2de6ffd640 10 monclient(hunting): get_auth_request con 0x7f2de80a99d0 auth_method 0 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2de6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2de6ffd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2de77fe640 10 monclient(hunting): get_auth_request con 0x7f2de80b8a90 auth_method 0 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2de77fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2dde7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.521+0000 7f2dde7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.522+0000 7f2dde7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:25.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.522+0000 7f2de6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:25.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.522+0000 7f2de6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:25.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.522+0000 7f2de6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:25.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.522+0000 7f2de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:25.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.522+0000 7f2de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:25.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.522+0000 7f2de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.524+0000 7f2de6ffd640 10 monclient(hunting): handle_auth_done global_id 17663 payload 824 2024-05-15T13:10:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.525+0000 7f2de6ffd640 1 -- 172.21.15.49:0/1977292532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de80b8a90 msgr2=0x7f2de81d8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.525+0000 7f2de6ffd640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de80b8a90 0x7f2de81d8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.525+0000 7f2de6ffd640 1 -- 172.21.15.49:0/1977292532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de80b1ef0 msgr2=0x7f2de813b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.525+0000 7f2de6ffd640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de80b1ef0 0x7f2de813b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.525+0000 7f2de6ffd640 10 monclient: _finish_hunting 0 2024-05-15T13:10:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.525+0000 7f2de6ffd640 1 monclient: found mon.noname-c 2024-05-15T13:10:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.525+0000 7f2de6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.525+0000 7f2de6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T13:10:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.525+0000 7f2de6ffd640 1 -- 172.21.15.49:0/1977292532 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de81d7a40 con 0x7f2de80a99d0 2024-05-15T13:10:25.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.527+0000 7f2de6ffd640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 0x7f2de813aa80 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800e870 tx=0x7f2dd800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:25.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.527+0000 7f2ddffff640 10 client.?.objecter ms_handle_connect 0x7f2de80a99d0 2024-05-15T13:10:25.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.527+0000 7f2ddffff640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 1 -- 172.21.15.49:0/1977292532 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8017070 con 0x7f2de80a99d0 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 10 monclient: dump: 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:25.522 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 10 monclient: _finish_auth 0 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 10 monclient: _check_auth_tickets 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 1 -- 172.21.15.49:0/1977292532 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd800ee60 con 0x7f2de80a99d0 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ded5d1640 5 monclient: authenticate success, global_id 17663 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ded5d1640 10 monclient: _renew_subs 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ded5d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de81d8780 con 0x7f2de80a99d0 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 1 -- 172.21.15.49:0/1977292532 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd800cb60 con 0x7f2de80a99d0 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ddffff640 10 monclient: dump: 2024-05-15T13:10:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2de5ffb640 4 set_mon_vals no callback set 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ded5d1640 10 client.17663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ded5d1640 10 monclient: _renew_subs 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ded5d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:25.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de813bfd0 con 0x7f2de80a99d0 2024-05-15T13:10:25.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.531+0000 7f2ded5d1640 1 librados: init done 2024-05-15T13:10:25.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.532+0000 7f2ded5d1640 10 client.17663.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:25.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.532+0000 7f2ded5d1640 15 client.17663.objecter _session_command_op_assign -1 1 2024-05-15T13:10:25.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.532+0000 7f2ded5d1640 20 client.17663.objecter _get_session osd=-1 returning homeless 2024-05-15T13:10:25.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.532+0000 7f2ded5d1640 10 client.17663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:25.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.532+0000 7f2ded5d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:25.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.532+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2de804f0e0 con 0x7f2de80a99d0 2024-05-15T13:10:25.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.556+0000 7f2ddffff640 1 -- 172.21.15.49:0/1977292532 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2dd801a070 con 0x7f2de80a99d0 2024-05-15T13:10:25.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.557+0000 7f2ddffff640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2dc806ceb0 0x7f2dc806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.557+0000 7f2dde7fc640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2dc806ceb0 0x7f2dc806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.557+0000 7f2dde7fc640 10 monclient: get_auth_request con 0x7f2dc806ceb0 auth_method 0 2024-05-15T13:10:25.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.557+0000 7f2ddffff640 10 client.17663.objecter ms_handle_connect 0x7f2dc806ceb0 2024-05-15T13:10:25.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.557+0000 7f2dde7fc640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2dc806ceb0 0x7f2dc806f370 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2de8134de0 tx=0x7f2dd003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:25.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 1 -- 172.21.15.49:0/1977292532 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f2dd8056b50 con 0x7f2de80a99d0 2024-05-15T13:10:25.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 10 client.17663.objecter ms_dispatch 0x7f2de8000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:25.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 3 client.17663.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:25.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 3 client.17663.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:25.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 10 client.17663.objecter checking command 1 2024-05-15T13:10:25.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] conn(0x7f2dc80758a0 0x7f2dc8077d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 20 client.17663.objecter _get_session s=0x7f2dc80751d0 osd=7 3 2024-05-15T13:10:25.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 20 client.17663.objecter put_session s=0x7f2dc80751d0 osd=7 3 2024-05-15T13:10:25.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 15 client.17663.objecter _session_command_op_remove -1 1 2024-05-15T13:10:25.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 20 client.17663.objecter _get_session s=0x7f2dc80751d0 osd=7 3 2024-05-15T13:10:25.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 20 client.17663.objecter get_session s=0x7f2dc80751d0 osd=7 3 2024-05-15T13:10:25.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 15 client.17663.objecter _session_command_op_assign 7 1 2024-05-15T13:10:25.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 20 client.17663.objecter put_session s=0x7f2dc80751d0 osd=7 4 2024-05-15T13:10:25.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 10 client.17663.objecter _send_command 1 2024-05-15T13:10:25.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 1 -- 172.21.15.49:0/1977292532 --> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2dd800be50 con 0x7f2dc80758a0 2024-05-15T13:10:25.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 20 client.17663.objecter dump_active .. 0 homeless 2024-05-15T13:10:25.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 1 -- 172.21.15.49:0/1977292532 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_get_version_reply(handle=1 version=1251) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dd8054d50 con 0x7f2de80a99d0 2024-05-15T13:10:25.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2ddffff640 10 monclient: handle_get_version_reply finishing 1 version 1251 2024-05-15T13:10:25.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.560+0000 7f2de77fe640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] conn(0x7f2dc80758a0 0x7f2dc8077d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.561+0000 7f2de77fe640 10 monclient: get_auth_request con 0x7f2dc80758a0 auth_method 0 2024-05-15T13:10:25.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.563+0000 7f2de77fe640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] conn(0x7f2dc80758a0 0x7f2dc8077d40 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:25.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.563+0000 7f2ddffff640 10 client.17663.objecter ms_handle_connect 0x7f2dc80758a0 2024-05-15T13:10:25.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.588+0000 7f2ddffff640 1 -- 172.21.15.49:0/1977292532 <== osd.7 v2:172.21.15.88:6824/2037618931 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2dd800be50 con 0x7f2dc80758a0 2024-05-15T13:10:25.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.588+0000 7f2ddffff640 10 client.17663.objecter ms_dispatch 0x7f2de8000b90 command_reply(tid 1: 0 ) v1 2024-05-15T13:10:25.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.588+0000 7f2ddffff640 10 client.17663.objecter _finish_command 1 = system:0 2024-05-15T13:10:25.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.588+0000 7f2ddffff640 20 client.17663.objecter put_session s=0x7f2dc80751d0 osd=7 4 2024-05-15T13:10:25.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.588+0000 7f2ddffff640 15 client.17663.objecter _session_command_op_remove 7 1 2024-05-15T13:10:25.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 10 client.17663.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-15T13:10:25.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 15 client.17663.objecter _session_command_op_assign -1 2 2024-05-15T13:10:25.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 20 client.17663.objecter _get_session s=0x7f2dc80751d0 osd=7 3 2024-05-15T13:10:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 20 client.17663.objecter put_session s=0x7f2dc80751d0 osd=7 3 2024-05-15T13:10:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 20 client.17663.objecter _get_session s=0x7f2dc80751d0 osd=7 3 2024-05-15T13:10:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 15 client.17663.objecter _session_command_op_remove -1 2 2024-05-15T13:10:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 20 client.17663.objecter get_session s=0x7f2dc80751d0 osd=7 3 2024-05-15T13:10:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 15 client.17663.objecter _session_command_op_assign 7 2 2024-05-15T13:10:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 20 client.17663.objecter put_session s=0x7f2dc80751d0 osd=7 4 2024-05-15T13:10:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 10 client.17663.objecter _send_command 2 2024-05-15T13:10:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.602+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 --> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2de813eb40 con 0x7f2dc80758a0 2024-05-15T13:10:25.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ddffff640 1 -- 172.21.15.49:0/1977292532 <== osd.7 v2:172.21.15.88:6824/2037618931 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2de813eb40 con 0x7f2dc80758a0 2024-05-15T13:10:25.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ddffff640 10 client.17663.objecter ms_dispatch 0x7f2de8000b90 command_reply(tid 2: 0 ) v1 2024-05-15T13:10:25.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ddffff640 10 client.17663.objecter _finish_command 2 = system:0 2024-05-15T13:10:25.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ddffff640 20 client.17663.objecter put_session s=0x7f2dc80751d0 osd=7 4 2024-05-15T13:10:25.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ddffff640 15 client.17663.objecter _session_command_op_remove 7 2 2024-05-15T13:10:25.605 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-05-15T13:10:25.605 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-15T13:10:25.605 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-05-15T13:10:25.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 10 librados: watch_flush enter 2024-05-15T13:10:25.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 10 librados: watch_flush exit 2024-05-15T13:10:25.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 10 client.17663.objecter close_session for osd.7 2024-05-15T13:10:25.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 >> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] conn(0x7f2dc80758a0 msgr2=0x7f2dc8077d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.88:6824/2037618931,v1:172.21.15.88:6825/2037618931] conn(0x7f2dc80758a0 0x7f2dc8077d40 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 20 client.17663.objecter put_session s=0x7f2dc80751d0 osd=7 1 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 20 client.17663.objecter shutdown clearing up homeless session... 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 10 client.17663.objecter successfully canceled tick 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2dc806ceb0 msgr2=0x7f2dc806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f2dc806ceb0 0x7f2dc806f370 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2de8134de0 tx=0x7f2dd003c040 comp rx=0 tx=0).stop 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 10 monclient: shutdown 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 msgr2=0x7f2de813aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 --2- 172.21.15.49:0/1977292532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de80a99d0 0x7f2de813aa80 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800e870 tx=0x7f2dd800ed30 comp rx=0 tx=0).stop 2024-05-15T13:10:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2de77fe640 1 -- 172.21.15.49:0/1977292532 reap_dead start 2024-05-15T13:10:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 shutdown_connections 2024-05-15T13:10:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 >> 172.21.15.49:0/1977292532 conn(0x7f2de8067110 msgr2=0x7f2de813f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 shutdown_connections 2024-05-15T13:10:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 -- 172.21.15.49:0/1977292532 wait complete. 2024-05-15T13:10:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.615+0000 7f2ded5d1640 1 librados: shutdown 2024-05-15T13:10:25.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-15T13:10:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.731+0000 7f0f1e2cf640 10 monclient: get_monmap_and_config 2024-05-15T13:10:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.731+0000 7f0f1e2cf640 10 monclient: build_initial_monmap 2024-05-15T13:10:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.732+0000 7f0f1e2cf640 10 monclient: monmap: 2024-05-15T13:10:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:25.732523+0000 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:25.732523+0000 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:25.722 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.732+0000 7f0f1e2cf640 1 Processor -- start 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.732+0000 7f0f1e2cf640 1 -- start start 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.732+0000 7f0f1e2cf640 10 monclient: init 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 0x7f0f1813f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 10 monclient(hunting): picked mon.noname-b con 0x7f0f180b5470 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0f1813fbb0 0x7f0f181426e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 10 monclient(hunting): picked mon.noname-c con 0x7f0f1813fbb0 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f18142c20 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1813fff0 con 0x7f0f18142c20 2024-05-15T13:10:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f180b5850 con 0x7f0f1813fbb0 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180aa420 con 0x7f0f180b5470 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.733+0000 7f0f1e2cf640 10 monclient(hunting): authenticate will time out at 3783.894978s 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f17fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 0x7f0f1813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f177fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0f1813fbb0 0x7f0f181426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f1c86c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0f18142c20 0x7f0f180b2bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:38174/0 (socket says 172.21.15.49:38174) 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f1c86c640 1 -- 172.21.15.49:0/3728317535 learned_addr learned my addr 172.21.15.49:0/3728317535 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f17fff640 10 monclient(hunting): get_auth_request con 0x7f0f180b5470 auth_method 0 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f17fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:25.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f177fe640 10 monclient(hunting): get_auth_request con 0x7f0f1813fbb0 auth_method 0 2024-05-15T13:10:25.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:25.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f177fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:25.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.734+0000 7f0f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:25.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.735+0000 7f0f17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:25.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.735+0000 7f0f17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:25.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.735+0000 7f0f17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:25.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.735+0000 7f0f177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:25.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.735+0000 7f0f177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:25.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.735+0000 7f0f177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.736+0000 7f0f17fff640 10 monclient(hunting): handle_auth_done global_id 17666 payload 274 2024-05-15T13:10:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.736+0000 7f0f17fff640 1 -- 172.21.15.49:0/3728317535 >> v2:172.21.15.88:3300/0 conn(0x7f0f1813fbb0 msgr2=0x7f0f181426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.736+0000 7f0f17fff640 1 --2- 172.21.15.49:0/3728317535 >> v2:172.21.15.88:3300/0 conn(0x7f0f1813fbb0 0x7f0f181426e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.736+0000 7f0f17fff640 1 -- 172.21.15.49:0/3728317535 >> v1:172.21.15.88:6789/0 conn(0x7f0f18142c20 legacy=0x7f0f180b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.736+0000 7f0f17fff640 10 monclient: _finish_hunting 0 2024-05-15T13:10:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.736+0000 7f0f17fff640 1 monclient: found mon.noname-b 2024-05-15T13:10:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.736+0000 7f0f17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.736+0000 7f0f17fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T13:10:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.736+0000 7f0f17fff640 1 -- 172.21.15.49:0/3728317535 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180ab420 con 0x7f0f180b5470 2024-05-15T13:10:25.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.738+0000 7f0f17fff640 1 --2- 172.21.15.49:0/3728317535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 0x7f0f1813f670 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f0f080097a0 tx=0x7f0f0802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=852811edfec44ced server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 1 -- 172.21.15.49:0/3728317535 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08035020 con 0x7f0f180b5470 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 10 monclient: dump: 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 10 monclient: _finish_auth 0 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 10 monclient: _check_auth_tickets 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 1 -- 172.21.15.49:0/3728317535 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f08031830 con 0x7f0f180b5470 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 1 -- 172.21.15.49:0/3728317535 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08031c00 con 0x7f0f180b5470 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f16ffd640 10 monclient: dump: 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:25.733 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f1e2cf640 5 monclient: authenticate success, global_id 17666 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f1e2cf640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:25.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.742+0000 7f0f1e2cf640 4 set_mon_vals no callback set 2024-05-15T13:10:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 10 monclient: shutdown 2024-05-15T13:10:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/3728317535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 msgr2=0x7f0f1813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 1 --2- 172.21.15.49:0/3728317535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 0x7f0f1813f670 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f0f080097a0 tx=0x7f0f0802b8d0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/3728317535 shutdown_connections 2024-05-15T13:10:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 1 --2- 172.21.15.49:0/3728317535 >> v2:172.21.15.88:3300/0 conn(0x7f0f1813fbb0 0x7f0f181426e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 1 --2- 172.21.15.49:0/3728317535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 0x7f0f1813f670 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/3728317535 >> 172.21.15.49:0/3728317535 conn(0x7f0f18066f00 msgr2=0x7f0f18067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/3728317535 shutdown_connections 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/3728317535 wait complete. 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 10 monclient: build_initial_monmap 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.743+0000 7f0f1e2cf640 10 monclient: monmap: 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.744+0000 7f0f1e2cf640 1 librados: starting msgr at 2024-05-15T13:10:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.744+0000 7f0f1e2cf640 1 librados: starting objecter 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.744+0000 7f0f1e2cf640 1 Processor -- start 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.744+0000 7f0f1e2cf640 1 -- start start 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.744+0000 7f0f1e2cf640 1 librados: setting wanted keys 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.744+0000 7f0f1e2cf640 1 librados: calling monclient init 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.744+0000 7f0f1e2cf640 10 monclient: init 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 0x7f0f180ad3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 10 monclient(hunting): picked mon.noname-c con 0x7f0f180b5470 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f1813fbb0 0x7f0f180acb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 10 monclient(hunting): picked mon.noname-a con 0x7f0f1813fbb0 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f18142c20 0x7f0f181d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 10 monclient(hunting): picked mon.noname-b con 0x7f0f18142c20 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180a9f00 con 0x7f0f1813fbb0 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18140220 con 0x7f0f18142c20 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180b2790 con 0x7f0f180b5470 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1e2cf640 10 monclient(hunting): authenticate will time out at 3783.906786s 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f1c86c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f18142c20 0x7f0f181d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f177fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f1813fbb0 0x7f0f180acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.745+0000 7f0f17fff640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 0x7f0f180ad3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f1c86c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f18142c20 0x7f0f181d7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35958/0 (socket says 172.21.15.49:35958) 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f1c86c640 1 -- 172.21.15.49:0/4273126992 learned_addr learned my addr 172.21.15.49:0/4273126992 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f1c86c640 10 monclient(hunting): get_auth_request con 0x7f0f18142c20 auth_method 0 2024-05-15T13:10:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f1c86c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f1c86c640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f1c86c640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f177fe640 10 monclient(hunting): get_auth_request con 0x7f0f1813fbb0 auth_method 0 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f177fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f17fff640 10 monclient(hunting): get_auth_request con 0x7f0f180b5470 auth_method 0 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f17fff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f1c86c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f1c86c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f1c86c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:25.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:25.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:25.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:25.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:25.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:25.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.746+0000 7f0f17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.749+0000 7f0f1c86c640 10 monclient(hunting): handle_auth_done global_id 22093 payload 824 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.749+0000 7f0f1c86c640 1 -- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 msgr2=0x7f0f180ad3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.749+0000 7f0f1c86c640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 0x7f0f180ad3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.749+0000 7f0f1c86c640 1 -- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f1813fbb0 msgr2=0x7f0f180acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.749+0000 7f0f1c86c640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f1813fbb0 0x7f0f180acb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.749+0000 7f0f1c86c640 10 monclient: _finish_hunting 0 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.749+0000 7f0f1c86c640 1 monclient: found mon.noname-b 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.749+0000 7f0f1c86c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.749+0000 7f0f1c86c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.749+0000 7f0f1c86c640 1 -- 172.21.15.49:0/4273126992 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180aec50 con 0x7f0f18142c20 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.751+0000 7f0f1c86c640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f18142c20 0x7f0f181d7c20 secure :-1 s=READY pgs=5475 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000ca70 tx=0x7f0f1000cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:25.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.751+0000 7f0f14ff9640 10 client.?.objecter ms_handle_connect 0x7f0f18142c20 2024-05-15T13:10:25.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.751+0000 7f0f14ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 1 -- 172.21.15.49:0/4273126992 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10017070 con 0x7f0f18142c20 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 10 monclient: dump: 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 10 monclient: _finish_auth 0 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 10 monclient: _check_auth_tickets 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 1 -- 172.21.15.49:0/4273126992 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1000abd0 con 0x7f0f18142c20 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 1 -- 172.21.15.49:0/4273126992 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10012400 con 0x7f0f18142c20 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f14ff9640 10 monclient: dump: 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f1e2cf640 5 monclient: authenticate success, global_id 22093 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f1e2cf640 10 monclient: _renew_subs 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f1e2cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/4273126992 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f180ad080 con 0x7f0f18142c20 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.754+0000 7f0f167fc640 4 set_mon_vals no callback set 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.755+0000 7f0f1e2cf640 10 client.22093.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.755+0000 7f0f1e2cf640 10 monclient: _renew_subs 2024-05-15T13:10:25.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.755+0000 7f0f1e2cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:25.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.755+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/4273126992 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f181d8340 con 0x7f0f18142c20 2024-05-15T13:10:25.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.755+0000 7f0f1e2cf640 1 librados: init done 2024-05-15T13:10:25.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.755+0000 7f0f1e2cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:25.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.755+0000 7f0f1e2cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:25.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.755+0000 7f0f1e2cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:25.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.755+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/4273126992 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f180adf00 con 0x7f0f18142c20 2024-05-15T13:10:25.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.773+0000 7f0f14ff9640 1 -- 172.21.15.49:0/4273126992 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0f1001a070 con 0x7f0f18142c20 2024-05-15T13:10:25.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.774+0000 7f0f14ff9640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f0f0406ceb0 0x7f0f0406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:25.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.774+0000 7f0f17fff640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f0f0406ceb0 0x7f0f0406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:25.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.774+0000 7f0f17fff640 10 monclient: get_auth_request con 0x7f0f0406ceb0 auth_method 0 2024-05-15T13:10:25.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.774+0000 7f0f14ff9640 10 client.22093.objecter ms_handle_connect 0x7f0f0406ceb0 2024-05-15T13:10:25.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.774+0000 7f0f17fff640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f0f0406ceb0 0x7f0f0406f370 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0f08036000 tx=0x7f0f08008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:25.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.776+0000 7f0f14ff9640 1 -- 172.21.15.49:0/4273126992 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f0f10056b40 con 0x7f0f18142c20 2024-05-15T13:10:25.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.776+0000 7f0f14ff9640 10 client.22093.objecter ms_dispatch 0x7f0f18000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:25.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.776+0000 7f0f14ff9640 3 client.22093.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:25.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.776+0000 7f0f14ff9640 3 client.22093.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:25.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:25.776+0000 7f0f14ff9640 20 client.22093.objecter dump_active .. 0 homeless 2024-05-15T13:10:26.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.022+0000 7f0f14ff9640 1 -- 172.21.15.49:0/4273126992 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0f10021020 con 0x7f0f18142c20 2024-05-15T13:10:26.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.022+0000 7f0f14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:26.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.022+0000 7f0f14ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:26.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.215+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/4273126992 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f18067730 con 0x7f0f0406ceb0 2024-05-15T13:10:26.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.215+0000 7f0f14ff9640 1 -- 172.21.15.49:0/4273126992 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0f18067730 con 0x7f0f0406ceb0 2024-05-15T13:10:26.205 INFO:teuthology.orchestra.run.smithi049.stderr:instructed osd(s) 7 to deep-scrub 2024-05-15T13:10:26.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 10 librados: watch_flush enter 2024-05-15T13:10:26.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 10 librados: watch_flush exit 2024-05-15T13:10:26.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 20 client.22093.objecter shutdown clearing up homeless session... 2024-05-15T13:10:26.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 10 client.22093.objecter successfully canceled tick 2024-05-15T13:10:26.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f0f0406ceb0 msgr2=0x7f0f0406f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:26.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f0f0406ceb0 0x7f0f0406f370 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0f08036000 tx=0x7f0f08008030 comp rx=0 tx=0).stop 2024-05-15T13:10:26.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 10 monclient: shutdown 2024-05-15T13:10:26.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/4273126992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f18142c20 msgr2=0x7f0f181d7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f18142c20 0x7f0f181d7c20 secure :-1 s=READY pgs=5475 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000ca70 tx=0x7f0f1000cf30 comp rx=0 tx=0).stop 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.217+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/4273126992 shutdown_connections 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.218+0000 7f0f1e2cf640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f0f0406ceb0 0x7f0f0406f370 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.218+0000 7f0f1e2cf640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f18142c20 0x7f0f181d7c20 unknown :-1 s=CLOSED pgs=5475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.218+0000 7f0f1e2cf640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f1813fbb0 0x7f0f180acb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.218+0000 7f0f1e2cf640 1 --2- 172.21.15.49:0/4273126992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180b5470 0x7f0f180ad3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.218+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/4273126992 >> 172.21.15.49:0/4273126992 conn(0x7f0f18066f00 msgr2=0x7f0f18069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.218+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/4273126992 shutdown_connections 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.218+0000 7f0f1e2cf640 1 -- 172.21.15.49:0/4273126992 wait complete. 2024-05-15T13:10:26.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.218+0000 7f0f1e2cf640 1 librados: shutdown 2024-05-15T13:10:26.221 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T13:10:26.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.337+0000 7f1a65347640 10 monclient: get_monmap_and_config 2024-05-15T13:10:26.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.337+0000 7f1a65347640 10 monclient: build_initial_monmap 2024-05-15T13:10:26.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.337+0000 7f1a65347640 10 monclient: monmap: 2024-05-15T13:10:26.327 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:26.327 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:26.327 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:26.338244+0000 2024-05-15T13:10:26.328 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:26.338244+0000 2024-05-15T13:10:26.328 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:26.328 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:26.328 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:26.328 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:26.328 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:26.328 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:26.328 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:26.328 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:26.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.338+0000 7f1a65347640 1 Processor -- start 2024-05-15T13:10:26.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.338+0000 7f1a65347640 1 -- start start 2024-05-15T13:10:26.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.338+0000 7f1a65347640 10 monclient: init 2024-05-15T13:10:26.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.338+0000 7f1a65347640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.338+0000 7f1a65347640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1a600b53e0 0x7f1a600a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 10 monclient(hunting): picked mon.noname-a con 0x7f1a600b53e0 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a60142870 0x7f1a60142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 10 monclient(hunting): picked mon.noname-c con 0x7f1a60142870 addr v2:172.21.15.88:3300/0 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a601431f0 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a600a9f30 con 0x7f1a601431f0 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a600b57c0 con 0x7f1a600b53e0 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a600aa230 con 0x7f1a60142870 2024-05-15T13:10:26.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a65347640 10 monclient(hunting): authenticate will time out at 3784.500512s 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a5effd640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1a600b53e0 0x7f1a600a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a5effd640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1a600b53e0 0x7f1a600a99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60896/0 (socket says 172.21.15.49:60896) 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a5effd640 1 -- 172.21.15.49:0/260665281 learned_addr learned my addr 172.21.15.49:0/260665281 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a5effd640 10 monclient(hunting): get_auth_request con 0x7f1a600b53e0 auth_method 0 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a5effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a5effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.339+0000 7f1a5e7fc640 1 --2- 172.21.15.49:0/260665281 >> v2:172.21.15.88:3300/0 conn(0x7f1a60142870 0x7f1a60142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.340+0000 7f1a5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.340+0000 7f1a5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.340+0000 7f1a5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.340+0000 7f1a5e7fc640 10 monclient(hunting): get_auth_request con 0x7f1a60142870 auth_method 0 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.340+0000 7f1a5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:26.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.340+0000 7f1a5e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:26.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.340+0000 7f1a5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:26.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.341+0000 7f1a5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:26.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.341+0000 7f1a5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:26.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.341+0000 7f1a5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:26.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.342+0000 7f1a5effd640 10 monclient(hunting): handle_auth_done global_id 16239 payload 274 2024-05-15T13:10:26.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.342+0000 7f1a5effd640 1 -- 172.21.15.49:0/260665281 >> v2:172.21.15.88:3300/0 conn(0x7f1a60142870 msgr2=0x7f1a60142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-15T13:10:26.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.342+0000 7f1a5effd640 1 --2- 172.21.15.49:0/260665281 >> v2:172.21.15.88:3300/0 conn(0x7f1a60142870 0x7f1a60142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.342+0000 7f1a5effd640 1 -- 172.21.15.49:0/260665281 >> v1:172.21.15.88:6789/0 conn(0x7f1a601431f0 legacy=0x7f1a600b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:26.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.342+0000 7f1a5effd640 10 monclient: _finish_hunting 0 2024-05-15T13:10:26.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.342+0000 7f1a5effd640 1 monclient: found mon.noname-a 2024-05-15T13:10:26.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.342+0000 7f1a5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:26.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.342+0000 7f1a5effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T13:10:26.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.342+0000 7f1a5effd640 1 -- 172.21.15.49:0/260665281 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a600ab230 con 0x7f1a600b53e0 2024-05-15T13:10:26.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.344+0000 7f1a5effd640 1 --2- 172.21.15.49:0/260665281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a600b53e0 0x7f1a600a99f0 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7f1a600b4660 tx=0x7f1a5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930e18a938b43fe2 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:26.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 1 -- 172.21.15.49:0/260665281 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50035020 con 0x7f1a600b53e0 2024-05-15T13:10:26.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 10 monclient: dump: 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 10 monclient: _finish_auth 0 2024-05-15T13:10:26.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 10 monclient: _check_auth_tickets 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 1 -- 172.21.15.49:0/260665281 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a50031870 con 0x7f1a600b53e0 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 5 monclient: authenticate success, global_id 16239 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 4 set_mon_vals no callback set 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 1 -- 172.21.15.49:0/260665281 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50031c40 con 0x7f1a600b53e0 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a5dffb640 10 monclient: dump: 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:26.340 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 10 monclient: shutdown 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 1 -- 172.21.15.49:0/260665281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a600b53e0 msgr2=0x7f1a600a99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 1 --2- 172.21.15.49:0/260665281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a600b53e0 0x7f1a600a99f0 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7f1a600b4660 tx=0x7f1a5002b910 comp rx=0 tx=0).stop 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 1 -- 172.21.15.49:0/260665281 shutdown_connections 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 1 --2- 172.21.15.49:0/260665281 >> v2:172.21.15.88:3300/0 conn(0x7f1a60142870 0x7f1a60142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 1 --2- 172.21.15.49:0/260665281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a600b53e0 0x7f1a600a99f0 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 1 -- 172.21.15.49:0/260665281 >> 172.21.15.49:0/260665281 conn(0x7f1a60066ed0 msgr2=0x7f1a600672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:26.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.348+0000 7f1a65347640 1 -- 172.21.15.49:0/260665281 shutdown_connections 2024-05-15T13:10:26.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.349+0000 7f1a65347640 1 -- 172.21.15.49:0/260665281 wait complete. 2024-05-15T13:10:26.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.349+0000 7f1a65347640 10 monclient: build_initial_monmap 2024-05-15T13:10:26.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.349+0000 7f1a65347640 10 monclient: monmap: 2024-05-15T13:10:26.347 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:26.347 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:26.347 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:26.348 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:26.348 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:26.348 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:26.348 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:26.348 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:26.348 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:26.349 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:26.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.349+0000 7f1a65347640 1 librados: starting msgr at 2024-05-15T13:10:26.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.349+0000 7f1a65347640 1 librados: starting objecter 2024-05-15T13:10:26.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.350+0000 7f1a65347640 1 Processor -- start 2024-05-15T13:10:26.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.350+0000 7f1a65347640 1 -- start start 2024-05-15T13:10:26.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.350+0000 7f1a65347640 1 librados: setting wanted keys 2024-05-15T13:10:26.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.350+0000 7f1a65347640 1 librados: calling monclient init 2024-05-15T13:10:26.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.350+0000 7f1a65347640 10 monclient: init 2024-05-15T13:10:26.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:26.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-15T13:10:26.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a600b53e0 0x7f1a601db370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:26.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 10 monclient(hunting): picked mon.noname-c con 0x7f1a600b53e0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:26.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a60142870 0x7f1a600b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:26.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 10 monclient(hunting): picked mon.noname-a con 0x7f1a60142870 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:26.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a601431f0 0x7f1a600b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:26.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 10 monclient(hunting): picked mon.noname-b con 0x7f1a601431f0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:26.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:26.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a600b3440 con 0x7f1a60142870 2024-05-15T13:10:26.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:26.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a600b32c0 con 0x7f1a601431f0 2024-05-15T13:10:26.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:26.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a600b35c0 con 0x7f1a600b53e0 2024-05-15T13:10:26.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:26.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a65347640 10 monclient(hunting): authenticate will time out at 3784.512823s 2024-05-15T13:10:26.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a5effd640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a600b53e0 0x7f1a601db370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:26.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a5e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a60142870 0x7f1a600b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:26.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.351+0000 7f1a5f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a601431f0 0x7f1a600b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:26.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a601431f0 0x7f1a600b0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.49:35970/0 (socket says 172.21.15.49:35970) 2024-05-15T13:10:26.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5f7fe640 1 -- 172.21.15.49:0/3939807817 learned_addr learned my addr 172.21.15.49:0/3939807817 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:26.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5f7fe640 10 monclient(hunting): get_auth_request con 0x7f1a601431f0 auth_method 0 2024-05-15T13:10:26.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:26.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:26.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:26.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5e7fc640 10 monclient(hunting): get_auth_request con 0x7f1a60142870 auth_method 0 2024-05-15T13:10:26.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:26.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:26.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:26.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5effd640 10 monclient(hunting): get_auth_request con 0x7f1a600b53e0 auth_method 0 2024-05-15T13:10:26.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:26.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5effd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:26.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:26.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:26.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:26.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:26.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:26.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:26.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:26.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:26.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:26.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.352+0000 7f1a5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:26.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.355+0000 7f1a5f7fe640 10 monclient(hunting): handle_auth_done global_id 22102 payload 824 2024-05-15T13:10:26.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.355+0000 7f1a5f7fe640 1 -- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a600b53e0 msgr2=0x7f1a601db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:26.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.355+0000 7f1a5f7fe640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a600b53e0 0x7f1a601db370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.355+0000 7f1a5f7fe640 1 -- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a60142870 msgr2=0x7f1a600b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:26.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.355+0000 7f1a5f7fe640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a60142870 0x7f1a600b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.355+0000 7f1a5f7fe640 10 monclient: _finish_hunting 0 2024-05-15T13:10:26.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.355+0000 7f1a5f7fe640 1 monclient: found mon.noname-b 2024-05-15T13:10:26.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.355+0000 7f1a5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:26.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.355+0000 7f1a5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3300/0 2024-05-15T13:10:26.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.355+0000 7f1a5f7fe640 1 -- 172.21.15.49:0/3939807817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a601db8b0 con 0x7f1a601431f0 2024-05-15T13:10:26.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.357+0000 7f1a5f7fe640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a601431f0 0x7f1a600b0c20 secure :-1 s=READY pgs=5477 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800c6b0 tx=0x7f1a5800cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:26.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.357+0000 7f1a477fe640 10 client.?.objecter ms_handle_connect 0x7f1a601431f0 2024-05-15T13:10:26.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.357+0000 7f1a477fe640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:26.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.360+0000 7f1a477fe640 1 -- 172.21.15.49:0/3939807817 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a58003b70 con 0x7f1a601431f0 2024-05-15T13:10:26.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.360+0000 7f1a477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:26.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.360+0000 7f1a477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:26.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.360+0000 7f1a477fe640 10 monclient: dump: 2024-05-15T13:10:26.362 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:26.363 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:26.363 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:26.363 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:26.363 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:26.363 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:26.363 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:26.364 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:26.364 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:26.364 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:26.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.360+0000 7f1a477fe640 10 monclient: _finish_auth 0 2024-05-15T13:10:26.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.360+0000 7f1a477fe640 10 monclient: _check_auth_tickets 2024-05-15T13:10:26.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.360+0000 7f1a477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:26.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a477fe640 1 -- 172.21.15.49:0/3939807817 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a58003d30 con 0x7f1a601431f0 2024-05-15T13:10:26.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a65347640 5 monclient: authenticate success, global_id 22102 2024-05-15T13:10:26.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a65347640 10 monclient: _renew_subs 2024-05-15T13:10:26.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a65347640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:26.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a65347640 1 -- 172.21.15.49:0/3939807817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6013ac50 con 0x7f1a601431f0 2024-05-15T13:10:26.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a477fe640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:26.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a477fe640 1 -- 172.21.15.49:0/3939807817 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a58005310 con 0x7f1a601431f0 2024-05-15T13:10:26.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:26.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-15T13:10:26.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a477fe640 10 monclient: dump: 2024-05-15T13:10:26.366 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:26.367 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:26.367 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:26.367 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:26.367 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:26.367 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:26.367 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:26.368 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:26.368 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:26.368 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:26.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a5d7fa640 4 set_mon_vals no callback set 2024-05-15T13:10:26.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a65347640 10 client.22102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:26.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a65347640 10 monclient: _renew_subs 2024-05-15T13:10:26.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a65347640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:26.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a65347640 1 -- 172.21.15.49:0/3939807817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6013a1b0 con 0x7f1a601431f0 2024-05-15T13:10:26.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.361+0000 7f1a65347640 1 librados: init done 2024-05-15T13:10:26.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.362+0000 7f1a65347640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:26.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.362+0000 7f1a65347640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:26.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.362+0000 7f1a65347640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3300/0 2024-05-15T13:10:26.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.362+0000 7f1a65347640 1 -- 172.21.15.49:0/3939807817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6004f0e0 con 0x7f1a601431f0 2024-05-15T13:10:26.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.613+0000 7f1a477fe640 1 -- 172.21.15.49:0/3939807817 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1a580054b0 con 0x7f1a601431f0 2024-05-15T13:10:26.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.613+0000 7f1a477fe640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f1a4c06ce60 0x7f1a4c06f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:26.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.613+0000 7f1a5effd640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f1a4c06ce60 0x7f1a4c06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:26.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.613+0000 7f1a5effd640 10 monclient: get_auth_request con 0x7f1a4c06ce60 auth_method 0 2024-05-15T13:10:26.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.614+0000 7f1a5effd640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f1a4c06ce60 0x7f1a4c06f320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1a50036000 tx=0x7f1a5003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:26.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.614+0000 7f1a477fe640 10 client.22102.objecter ms_handle_connect 0x7f1a4c06ce60 2024-05-15T13:10:26.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.616+0000 7f1a477fe640 1 -- 172.21.15.49:0/3939807817 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f1a580057a0 con 0x7f1a601431f0 2024-05-15T13:10:26.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.616+0000 7f1a477fe640 10 client.22102.objecter ms_dispatch 0x7f1a60000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:26.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.616+0000 7f1a477fe640 3 client.22102.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:26.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.616+0000 7f1a477fe640 3 client.22102.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:26.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.616+0000 7f1a477fe640 20 client.22102.objecter dump_active .. 0 homeless 2024-05-15T13:10:26.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.626+0000 7f1a477fe640 1 -- 172.21.15.49:0/3939807817 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1a58059270 con 0x7f1a601431f0 2024-05-15T13:10:26.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.626+0000 7f1a477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:26.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.626+0000 7f1a477fe640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:26.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.810+0000 7f1a65347640 1 -- 172.21.15.49:0/3939807817 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1a600b1340 con 0x7f1a4c06ce60 2024-05-15T13:10:26.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.815+0000 7f1a477fe640 1 -- 172.21.15.49:0/3939807817 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+196684 (secure 0 0 0) 0x7f1a50078060 con 0x7f1a4c06ce60 2024-05-15T13:10:26.806 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T13:10:26.807 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-05-15T13:10:26.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 10 librados: watch_flush enter 2024-05-15T13:10:26.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 10 librados: watch_flush exit 2024-05-15T13:10:26.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 20 client.22102.objecter shutdown clearing up homeless session... 2024-05-15T13:10:26.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 10 client.22102.objecter successfully canceled tick 2024-05-15T13:10:26.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 -- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f1a4c06ce60 msgr2=0x7f1a4c06f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f1a4c06ce60 0x7f1a4c06f320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1a50036000 tx=0x7f1a5003c040 comp rx=0 tx=0).stop 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 10 monclient: shutdown 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 -- 172.21.15.49:0/3939807817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a601431f0 msgr2=0x7f1a600b0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a601431f0 0x7f1a600b0c20 secure :-1 s=READY pgs=5477 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800c6b0 tx=0x7f1a5800cb70 comp rx=0 tx=0).stop 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 -- 172.21.15.49:0/3939807817 shutdown_connections 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f1a4c06ce60 0x7f1a4c06f320 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a601431f0 0x7f1a600b0c20 unknown :-1 s=CLOSED pgs=5477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a60142870 0x7f1a600b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 --2- 172.21.15.49:0/3939807817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a600b53e0 0x7f1a601db370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 -- 172.21.15.49:0/3939807817 >> 172.21.15.49:0/3939807817 conn(0x7f1a60066ed0 msgr2=0x7f1a6013dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 -- 172.21.15.49:0/3939807817 shutdown_connections 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 -- 172.21.15.49:0/3939807817 wait complete. 2024-05-15T13:10:26.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:26.817+0000 7f1a65347640 1 librados: shutdown 2024-05-15T13:10:26.818 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2393,"stamp":"2024-05-15T13:10:25.053209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465988,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":170,"num_write_kb":1429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":577304,"kb_used_data":43272,"kb_used_omap":12,"kb_used_meta":533939,"kb_avail":754397416,"statfs":{"total":773094113280,"available":772502953984,"internally_reserved":0,"allocated":44310528,"data_stored":22861464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":546754096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.989995"},"pg_stats":[{"pgid":"64.1f","version":"189'3","reported_seq":2379,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.725023+0000","last_change":"2024-05-15T12:35:50.056262+0000","last_active":"2024-05-15T13:10:17.725023+0000","last_peered":"2024-05-15T13:10:17.725023+0000","last_clean":"2024-05-15T13:10:17.725023+0000","last_became_active":"2024-05-15T12:35:49.922024+0000","last_became_peered":"2024-05-15T12:35:49.922024+0000","last_unstale":"2024-05-15T13:10:17.725023+0000","last_undegraded":"2024-05-15T13:10:17.725023+0000","last_fullsized":"2024-05-15T13:10:17.725023+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:51:48.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1e","version":"0'0","reported_seq":2374,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.658640+0000","last_change":"2024-05-15T12:35:32.331096+0000","last_active":"2024-05-15T13:10:17.658640+0000","last_peered":"2024-05-15T13:10:17.658640+0000","last_clean":"2024-05-15T13:10:17.658640+0000","last_became_active":"2024-05-15T12:35:32.314677+0000","last_became_peered":"2024-05-15T12:35:32.314677+0000","last_unstale":"2024-05-15T13:10:17.658640+0000","last_undegraded":"2024-05-15T13:10:17.658640+0000","last_fullsized":"2024-05-15T13:10:17.658640+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:14:42.373968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1d","version":"190'2","reported_seq":2363,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.729279+0000","last_change":"2024-05-15T12:36:16.192838+0000","last_active":"2024-05-15T13:10:17.729279+0000","last_peered":"2024-05-15T13:10:17.729279+0000","last_clean":"2024-05-15T13:10:17.729279+0000","last_became_active":"2024-05-15T12:36:16.182561+0000","last_became_peered":"2024-05-15T12:36:16.182561+0000","last_unstale":"2024-05-15T13:10:17.729279+0000","last_undegraded":"2024-05-15T13:10:17.729279+0000","last_fullsized":"2024-05-15T13:10:17.729279+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1c","version":"0'0","reported_seq":2375,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.688685+0000","last_change":"2024-05-15T12:35:47.611774+0000","last_active":"2024-05-15T13:10:17.688685+0000","last_peered":"2024-05-15T13:10:17.688685+0000","last_clean":"2024-05-15T13:10:17.688685+0000","last_became_active":"2024-05-15T12:35:47.561938+0000","last_became_peered":"2024-05-15T12:35:47.561938+0000","last_unstale":"2024-05-15T13:10:17.688685+0000","last_undegraded":"2024-05-15T13:10:17.688685+0000","last_fullsized":"2024-05-15T13:10:17.688685+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:53:18.771807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1b","version":"163'1","reported_seq":2384,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.711457+0000","last_change":"2024-05-15T12:35:31.343846+0000","last_active":"2024-05-15T13:10:18.711457+0000","last_peered":"2024-05-15T13:10:18.711457+0000","last_clean":"2024-05-15T13:10:18.711457+0000","last_became_active":"2024-05-15T12:35:31.337467+0000","last_became_peered":"2024-05-15T12:35:31.337467+0000","last_unstale":"2024-05-15T13:10:18.711457+0000","last_undegraded":"2024-05-15T13:10:18.711457+0000","last_fullsized":"2024-05-15T13:10:18.711457+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:43:08.695881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.1a","version":"190'4","reported_seq":2366,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.714635+0000","last_change":"2024-05-15T12:35:44.341452+0000","last_active":"2024-05-15T13:10:17.714635+0000","last_peered":"2024-05-15T13:10:17.714635+0000","last_clean":"2024-05-15T13:10:17.714635+0000","last_became_active":"2024-05-15T12:35:44.321742+0000","last_became_peered":"2024-05-15T12:35:44.321742+0000","last_unstale":"2024-05-15T13:10:17.714635+0000","last_undegraded":"2024-05-15T13:10:17.714635+0000","last_fullsized":"2024-05-15T13:10:17.714635+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.19","version":"190'1","reported_seq":2377,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.670090+0000","last_change":"2024-05-15T12:35:49.203007+0000","last_active":"2024-05-15T13:10:17.670090+0000","last_peered":"2024-05-15T13:10:17.670090+0000","last_clean":"2024-05-15T13:10:17.670090+0000","last_became_active":"2024-05-15T12:35:49.195589+0000","last_became_peered":"2024-05-15T12:35:49.195589+0000","last_unstale":"2024-05-15T13:10:17.670090+0000","last_undegraded":"2024-05-15T13:10:17.670090+0000","last_fullsized":"2024-05-15T13:10:17.670090+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:31:42.949909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.18","version":"165'3","reported_seq":2365,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.697393+0000","last_change":"2024-05-15T12:35:44.420062+0000","last_active":"2024-05-15T13:10:17.697393+0000","last_peered":"2024-05-15T13:10:17.697393+0000","last_clean":"2024-05-15T13:10:17.697393+0000","last_became_active":"2024-05-15T12:35:44.403250+0000","last_became_peered":"2024-05-15T12:35:44.403250+0000","last_unstale":"2024-05-15T13:10:17.697393+0000","last_undegraded":"2024-05-15T13:10:17.697393+0000","last_fullsized":"2024-05-15T13:10:17.697393+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.17","version":"190'2","reported_seq":2371,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.737510+0000","last_change":"2024-05-15T12:35:50.527941+0000","last_active":"2024-05-15T13:10:18.737510+0000","last_peered":"2024-05-15T13:10:18.737510+0000","last_clean":"2024-05-15T13:10:18.737510+0000","last_became_active":"2024-05-15T12:35:50.518230+0000","last_became_peered":"2024-05-15T12:35:50.518230+0000","last_unstale":"2024-05-15T13:10:18.737510+0000","last_undegraded":"2024-05-15T13:10:18.737510+0000","last_fullsized":"2024-05-15T13:10:18.737510+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:55:42.493548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.16","version":"180'4","reported_seq":2359,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.660534+0000","last_change":"2024-05-15T12:35:47.452786+0000","last_active":"2024-05-15T13:10:17.660534+0000","last_peered":"2024-05-15T13:10:17.660534+0000","last_clean":"2024-05-15T13:10:17.660534+0000","last_became_active":"2024-05-15T12:35:47.446210+0000","last_became_peered":"2024-05-15T12:35:47.446210+0000","last_unstale":"2024-05-15T13:10:17.660534+0000","last_undegraded":"2024-05-15T13:10:17.660534+0000","last_fullsized":"2024-05-15T13:10:17.660534+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:10:41.816557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.15","version":"190'2","reported_seq":2378,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.710805+0000","last_change":"2024-05-15T12:35:49.852126+0000","last_active":"2024-05-15T13:10:17.710805+0000","last_peered":"2024-05-15T13:10:17.710805+0000","last_clean":"2024-05-15T13:10:17.710805+0000","last_became_active":"2024-05-15T12:35:49.823395+0000","last_became_peered":"2024-05-15T12:35:49.823395+0000","last_unstale":"2024-05-15T13:10:17.710805+0000","last_undegraded":"2024-05-15T13:10:17.710805+0000","last_fullsized":"2024-05-15T13:10:17.710805+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:35:00.942178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.14","version":"0'0","reported_seq":2369,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.704485+0000","last_change":"2024-05-15T12:35:47.935648+0000","last_active":"2024-05-15T13:10:18.704485+0000","last_peered":"2024-05-15T13:10:18.704485+0000","last_clean":"2024-05-15T13:10:18.704485+0000","last_became_active":"2024-05-15T12:35:47.926796+0000","last_became_peered":"2024-05-15T12:35:47.926796+0000","last_unstale":"2024-05-15T13:10:18.704485+0000","last_undegraded":"2024-05-15T13:10:18.704485+0000","last_fullsized":"2024-05-15T13:10:18.704485+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:12:08.277421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.13","version":"171'1","reported_seq":2384,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.712495+0000","last_change":"2024-05-15T12:35:28.188222+0000","last_active":"2024-05-15T13:10:18.712495+0000","last_peered":"2024-05-15T13:10:18.712495+0000","last_clean":"2024-05-15T13:10:18.712495+0000","last_became_active":"2024-05-15T12:35:28.182751+0000","last_became_peered":"2024-05-15T12:35:28.182751+0000","last_unstale":"2024-05-15T13:10:18.712495+0000","last_undegraded":"2024-05-15T13:10:18.712495+0000","last_fullsized":"2024-05-15T13:10:18.712495+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:38:40.701230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.12","version":"79'1","reported_seq":2382,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.712348+0000","last_change":"2024-05-15T12:35:27.725919+0000","last_active":"2024-05-15T13:10:18.712348+0000","last_peered":"2024-05-15T13:10:18.712348+0000","last_clean":"2024-05-15T13:10:18.712348+0000","last_became_active":"2024-05-15T12:35:27.719548+0000","last_became_peered":"2024-05-15T12:35:27.719548+0000","last_unstale":"2024-05-15T13:10:18.712348+0000","last_undegraded":"2024-05-15T13:10:18.712348+0000","last_fullsized":"2024-05-15T13:10:18.712348+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:26:15.224906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.11","version":"172'1","reported_seq":2373,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.700406+0000","last_change":"2024-05-15T12:35:50.434980+0000","last_active":"2024-05-15T13:10:18.700406+0000","last_peered":"2024-05-15T13:10:18.700406+0000","last_clean":"2024-05-15T13:10:18.700406+0000","last_became_active":"2024-05-15T12:35:50.426351+0000","last_became_peered":"2024-05-15T12:35:50.426351+0000","last_unstale":"2024-05-15T13:10:18.700406+0000","last_undegraded":"2024-05-15T13:10:18.700406+0000","last_fullsized":"2024-05-15T13:10:18.700406+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:11:28.596763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.10","version":"190'3","reported_seq":2369,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.640579+0000","last_change":"2024-05-15T12:35:34.592833+0000","last_active":"2024-05-15T13:10:17.640579+0000","last_peered":"2024-05-15T13:10:17.640579+0000","last_clean":"2024-05-15T13:10:17.640579+0000","last_became_active":"2024-05-15T12:35:34.587282+0000","last_became_peered":"2024-05-15T12:35:34.587282+0000","last_unstale":"2024-05-15T13:10:17.640579+0000","last_undegraded":"2024-05-15T13:10:17.640579+0000","last_fullsized":"2024-05-15T13:10:17.640579+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:52:35.500859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.f","version":"155'2","reported_seq":2364,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.748369+0000","last_change":"2024-05-15T12:35:47.741135+0000","last_active":"2024-05-15T13:10:17.748369+0000","last_peered":"2024-05-15T13:10:17.748369+0000","last_clean":"2024-05-15T13:10:17.748369+0000","last_became_active":"2024-05-15T12:35:47.063782+0000","last_became_peered":"2024-05-15T12:35:47.063782+0000","last_unstale":"2024-05-15T13:10:17.748369+0000","last_undegraded":"2024-05-15T13:10:17.748369+0000","last_fullsized":"2024-05-15T13:10:17.748369+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.e","version":"174'1","reported_seq":2382,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.734183+0000","last_change":"2024-05-15T12:35:28.039423+0000","last_active":"2024-05-15T13:10:18.734183+0000","last_peered":"2024-05-15T13:10:18.734183+0000","last_clean":"2024-05-15T13:10:18.734183+0000","last_became_active":"2024-05-15T12:35:28.033694+0000","last_became_peered":"2024-05-15T12:35:28.033694+0000","last_unstale":"2024-05-15T13:10:18.734183+0000","last_undegraded":"2024-05-15T13:10:18.734183+0000","last_fullsized":"2024-05-15T13:10:18.734183+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:13:51.876959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.d","version":"175'3","reported_seq":2373,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.721988+0000","last_change":"2024-05-15T12:35:50.592101+0000","last_active":"2024-05-15T13:10:18.721988+0000","last_peered":"2024-05-15T13:10:18.721988+0000","last_clean":"2024-05-15T13:10:18.721988+0000","last_became_active":"2024-05-15T12:35:50.586234+0000","last_became_peered":"2024-05-15T12:35:50.586234+0000","last_unstale":"2024-05-15T13:10:18.721988+0000","last_undegraded":"2024-05-15T13:10:18.721988+0000","last_fullsized":"2024-05-15T13:10:18.721988+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:23:52.114251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.c","version":"190'1","reported_seq":2375,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.714788+0000","last_change":"2024-05-15T12:35:39.569380+0000","last_active":"2024-05-15T13:10:18.714788+0000","last_peered":"2024-05-15T13:10:18.714788+0000","last_clean":"2024-05-15T13:10:18.714788+0000","last_became_active":"2024-05-15T12:35:39.550911+0000","last_became_peered":"2024-05-15T12:35:39.550911+0000","last_unstale":"2024-05-15T13:10:18.714788+0000","last_undegraded":"2024-05-15T13:10:18.714788+0000","last_fullsized":"2024-05-15T13:10:18.714788+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.b","version":"186'2","reported_seq":2373,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.675934+0000","last_change":"2024-05-15T12:35:22.786550+0000","last_active":"2024-05-15T13:10:17.675934+0000","last_peered":"2024-05-15T13:10:17.675934+0000","last_clean":"2024-05-15T13:10:17.675934+0000","last_became_active":"2024-05-15T12:35:22.781126+0000","last_became_peered":"2024-05-15T12:35:22.781126+0000","last_unstale":"2024-05-15T13:10:17.675934+0000","last_undegraded":"2024-05-15T13:10:17.675934+0000","last_fullsized":"2024-05-15T13:10:17.675934+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:11:51.197065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.a","version":"170'1","reported_seq":2372,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.720534+0000","last_change":"2024-05-15T12:35:47.948588+0000","last_active":"2024-05-15T13:10:18.720534+0000","last_peered":"2024-05-15T13:10:18.720534+0000","last_clean":"2024-05-15T13:10:18.720534+0000","last_became_active":"2024-05-15T12:35:47.943168+0000","last_became_peered":"2024-05-15T12:35:47.943168+0000","last_unstale":"2024-05-15T13:10:18.720534+0000","last_undegraded":"2024-05-15T13:10:18.720534+0000","last_fullsized":"2024-05-15T13:10:18.720534+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:00:16.296438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.9","version":"190'1","reported_seq":2379,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.672170+0000","last_change":"2024-05-15T12:35:49.112002+0000","last_active":"2024-05-15T13:10:17.672170+0000","last_peered":"2024-05-15T13:10:17.672170+0000","last_clean":"2024-05-15T13:10:17.672170+0000","last_became_active":"2024-05-15T12:35:49.106669+0000","last_became_peered":"2024-05-15T12:35:49.106669+0000","last_unstale":"2024-05-15T13:10:17.672170+0000","last_undegraded":"2024-05-15T13:10:17.672170+0000","last_fullsized":"2024-05-15T13:10:17.672170+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:32:01.416983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.8","version":"0'0","reported_seq":2383,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.676244+0000","last_change":"2024-05-15T12:35:27.822672+0000","last_active":"2024-05-15T13:10:18.676244+0000","last_peered":"2024-05-15T13:10:18.676244+0000","last_clean":"2024-05-15T13:10:18.676244+0000","last_became_active":"2024-05-15T12:35:27.816334+0000","last_became_peered":"2024-05-15T12:35:27.816334+0000","last_unstale":"2024-05-15T13:10:18.676244+0000","last_undegraded":"2024-05-15T13:10:18.676244+0000","last_fullsized":"2024-05-15T13:10:18.676244+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:10:34.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.7","version":"172'1","reported_seq":2377,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.727087+0000","last_change":"2024-05-15T12:35:47.775023+0000","last_active":"2024-05-15T13:10:17.727087+0000","last_peered":"2024-05-15T13:10:17.727087+0000","last_clean":"2024-05-15T13:10:17.727087+0000","last_became_active":"2024-05-15T12:35:47.766045+0000","last_became_peered":"2024-05-15T12:35:47.766045+0000","last_unstale":"2024-05-15T13:10:17.727087+0000","last_undegraded":"2024-05-15T13:10:17.727087+0000","last_fullsized":"2024-05-15T13:10:17.727087+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:35:16.103745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.6","version":"0'0","reported_seq":2390,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:22.244389+0000","last_change":"2024-05-15T13:10:22.245242+0000","last_active":"2024-05-15T13:10:22.244389+0000","last_peered":"2024-05-15T13:10:22.244389+0000","last_clean":"2024-05-15T13:10:22.244389+0000","last_became_active":"2024-05-15T12:35:59.010795+0000","last_became_peered":"2024-05-15T12:35:59.010795+0000","last_unstale":"2024-05-15T13:10:22.244389+0000","last_undegraded":"2024-05-15T13:10:22.244389+0000","last_fullsized":"2024-05-15T13:10:22.244389+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:22.242692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:22.242692+0000","last_clean_scrub_stamp":"2024-05-15T13:10:22.242692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:07:11.477600+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.5","version":"161'1","reported_seq":2364,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.731622+0000","last_change":"2024-05-15T12:36:31.757984+0000","last_active":"2024-05-15T13:10:17.731622+0000","last_peered":"2024-05-15T13:10:17.731622+0000","last_clean":"2024-05-15T13:10:17.731622+0000","last_became_active":"2024-05-15T12:36:31.177992+0000","last_became_peered":"2024-05-15T12:36:31.177992+0000","last_unstale":"2024-05-15T13:10:17.731622+0000","last_undegraded":"2024-05-15T13:10:17.731622+0000","last_fullsized":"2024-05-15T13:10:17.731622+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.4","version":"0'0","reported_seq":2388,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:23.245685+0000","last_change":"2024-05-15T13:10:23.246607+0000","last_active":"2024-05-15T13:10:23.245685+0000","last_peered":"2024-05-15T13:10:23.245685+0000","last_clean":"2024-05-15T13:10:23.245685+0000","last_became_active":"2024-05-15T12:35:59.238121+0000","last_became_peered":"2024-05-15T12:35:59.238121+0000","last_unstale":"2024-05-15T13:10:23.245685+0000","last_undegraded":"2024-05-15T13:10:23.245685+0000","last_fullsized":"2024-05-15T13:10:23.245685+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:23.243967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:23.243967+0000","last_clean_scrub_stamp":"2024-05-15T13:10:23.243967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:02:22.323698+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.3","version":"190'4","reported_seq":2381,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.678474+0000","last_change":"2024-05-15T12:35:47.667458+0000","last_active":"2024-05-15T13:10:17.678474+0000","last_peered":"2024-05-15T13:10:17.678474+0000","last_clean":"2024-05-15T13:10:17.678474+0000","last_became_active":"2024-05-15T12:35:47.639452+0000","last_became_peered":"2024-05-15T12:35:47.639452+0000","last_unstale":"2024-05-15T13:10:17.678474+0000","last_undegraded":"2024-05-15T13:10:17.678474+0000","last_fullsized":"2024-05-15T13:10:17.678474+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:00:58.006820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":704,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.2","version":"1251'2","reported_seq":2403,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:21.345186+0000","last_change":"2024-05-15T13:10:21.346519+0000","last_active":"2024-05-15T13:10:21.345186+0000","last_peered":"2024-05-15T13:10:21.345186+0000","last_clean":"2024-05-15T13:10:21.345186+0000","last_became_active":"2024-05-15T12:35:32.699857+0000","last_became_peered":"2024-05-15T12:35:32.699857+0000","last_unstale":"2024-05-15T13:10:21.345186+0000","last_undegraded":"2024-05-15T13:10:21.345186+0000","last_fullsized":"2024-05-15T13:10:21.345186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:21.343475+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:21.343475+0000","last_clean_scrub_stamp":"2024-05-15T13:10:21.343475+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:44:28.279463+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1","version":"190'3","reported_seq":2367,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.692403+0000","last_change":"2024-05-15T12:36:13.620976+0000","last_active":"2024-05-15T13:10:17.692403+0000","last_peered":"2024-05-15T13:10:17.692403+0000","last_clean":"2024-05-15T13:10:17.692403+0000","last_became_active":"2024-05-15T12:36:13.614839+0000","last_became_peered":"2024-05-15T12:36:13.614839+0000","last_unstale":"2024-05-15T13:10:17.692403+0000","last_undegraded":"2024-05-15T13:10:17.692403+0000","last_fullsized":"2024-05-15T13:10:17.692403+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.0","version":"166'1","reported_seq":2360,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.645396+0000","last_change":"2024-05-15T12:35:47.277656+0000","last_active":"2024-05-15T13:10:17.645396+0000","last_peered":"2024-05-15T13:10:17.645396+0000","last_clean":"2024-05-15T13:10:17.645396+0000","last_became_active":"2024-05-15T12:35:47.271050+0000","last_became_peered":"2024-05-15T12:35:47.271050+0000","last_unstale":"2024-05-15T13:10:17.645396+0000","last_undegraded":"2024-05-15T13:10:17.645396+0000","last_fullsized":"2024-05-15T13:10:17.645396+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:35:14.324269+0000","last_clean_scrub_stamp":"2024-05-15T12:35:14.324269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:12:17.443633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":2406,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.746214+0000","last_change":"2024-05-15T12:34:57.169160+0000","last_active":"2024-05-15T13:10:17.746214+0000","last_peered":"2024-05-15T13:10:17.746214+0000","last_clean":"2024-05-15T13:10:17.746214+0000","last_became_active":"2024-05-15T12:34:57.154951+0000","last_became_peered":"2024-05-15T12:34:57.154951+0000","last_unstale":"2024-05-15T13:10:17.746214+0000","last_undegraded":"2024-05-15T13:10:17.746214+0000","last_fullsized":"2024-05-15T13:10:17.746214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":2408,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.725751+0000","last_change":"2024-05-15T12:34:54.507835+0000","last_active":"2024-05-15T13:10:17.725751+0000","last_peered":"2024-05-15T13:10:17.725751+0000","last_clean":"2024-05-15T13:10:17.725751+0000","last_became_active":"2024-05-15T12:34:54.498165+0000","last_became_peered":"2024-05-15T12:34:54.498165+0000","last_unstale":"2024-05-15T13:10:17.725751+0000","last_undegraded":"2024-05-15T13:10:17.725751+0000","last_fullsized":"2024-05-15T13:10:17.725751+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":2413,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.724164+0000","last_change":"2024-05-15T12:34:52.684394+0000","last_active":"2024-05-15T13:10:18.724164+0000","last_peered":"2024-05-15T13:10:18.724164+0000","last_clean":"2024-05-15T13:10:18.724164+0000","last_became_active":"2024-05-15T12:34:52.677394+0000","last_became_peered":"2024-05-15T12:34:52.677394+0000","last_unstale":"2024-05-15T13:10:18.724164+0000","last_undegraded":"2024-05-15T13:10:18.724164+0000","last_fullsized":"2024-05-15T13:10:18.724164+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:10:31.152782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":2419,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.690921+0000","last_change":"2024-05-15T12:34:50.121401+0000","last_active":"2024-05-15T13:10:17.690921+0000","last_peered":"2024-05-15T13:10:17.690921+0000","last_clean":"2024-05-15T13:10:17.690921+0000","last_became_active":"2024-05-15T12:34:50.114215+0000","last_became_peered":"2024-05-15T12:34:50.114215+0000","last_unstale":"2024-05-15T13:10:17.690921+0000","last_undegraded":"2024-05-15T13:10:17.690921+0000","last_fullsized":"2024-05-15T13:10:17.690921+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:04:16.442442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":2417,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.502841+0000","last_change":"2024-05-15T12:34:54.772141+0000","last_active":"2024-05-15T13:10:18.502841+0000","last_peered":"2024-05-15T13:10:18.502841+0000","last_clean":"2024-05-15T13:10:18.502841+0000","last_became_active":"2024-05-15T12:34:54.762972+0000","last_became_peered":"2024-05-15T12:34:54.762972+0000","last_unstale":"2024-05-15T13:10:18.502841+0000","last_undegraded":"2024-05-15T13:10:18.502841+0000","last_fullsized":"2024-05-15T13:10:18.502841+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:01:23.482313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":2421,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.722562+0000","last_change":"2024-05-15T12:34:34.684239+0000","last_active":"2024-05-15T13:10:18.722562+0000","last_peered":"2024-05-15T13:10:18.722562+0000","last_clean":"2024-05-15T13:10:18.722562+0000","last_became_active":"2024-05-15T12:34:34.670411+0000","last_became_peered":"2024-05-15T12:34:34.670411+0000","last_unstale":"2024-05-15T13:10:18.722562+0000","last_undegraded":"2024-05-15T13:10:18.722562+0000","last_fullsized":"2024-05-15T13:10:18.722562+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:15:28.642471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":2420,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.674247+0000","last_change":"2024-05-15T12:34:50.164567+0000","last_active":"2024-05-15T13:10:17.674247+0000","last_peered":"2024-05-15T13:10:17.674247+0000","last_clean":"2024-05-15T13:10:17.674247+0000","last_became_active":"2024-05-15T12:34:50.159068+0000","last_became_peered":"2024-05-15T12:34:50.159068+0000","last_unstale":"2024-05-15T13:10:17.674247+0000","last_undegraded":"2024-05-15T13:10:17.674247+0000","last_fullsized":"2024-05-15T13:10:17.674247+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:05:11.053203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":2426,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.656707+0000","last_change":"2024-05-15T12:34:39.918992+0000","last_active":"2024-05-15T13:10:17.656707+0000","last_peered":"2024-05-15T13:10:17.656707+0000","last_clean":"2024-05-15T13:10:17.656707+0000","last_became_active":"2024-05-15T12:34:39.907050+0000","last_became_peered":"2024-05-15T12:34:39.907050+0000","last_unstale":"2024-05-15T13:10:17.656707+0000","last_undegraded":"2024-05-15T13:10:17.656707+0000","last_fullsized":"2024-05-15T13:10:17.656707+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:10:12.279986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":2400,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.667321+0000","last_change":"2024-05-15T12:34:44.459281+0000","last_active":"2024-05-15T13:10:17.667321+0000","last_peered":"2024-05-15T13:10:17.667321+0000","last_clean":"2024-05-15T13:10:17.667321+0000","last_became_active":"2024-05-15T12:34:44.443445+0000","last_became_peered":"2024-05-15T12:34:44.443445+0000","last_unstale":"2024-05-15T13:10:17.667321+0000","last_undegraded":"2024-05-15T13:10:17.667321+0000","last_fullsized":"2024-05-15T13:10:17.667321+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:03:29.028978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":2402,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.664345+0000","last_change":"2024-05-15T12:34:38.200148+0000","last_active":"2024-05-15T13:10:17.664345+0000","last_peered":"2024-05-15T13:10:17.664345+0000","last_clean":"2024-05-15T13:10:17.664345+0000","last_became_active":"2024-05-15T12:34:38.194477+0000","last_became_peered":"2024-05-15T12:34:38.194477+0000","last_unstale":"2024-05-15T13:10:17.664345+0000","last_undegraded":"2024-05-15T13:10:17.664345+0000","last_fullsized":"2024-05-15T13:10:17.664345+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:28:19.294534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":2415,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:24.223236+0000","last_change":"2024-05-15T13:10:24.224104+0000","last_active":"2024-05-15T13:10:24.223236+0000","last_peered":"2024-05-15T13:10:24.223236+0000","last_clean":"2024-05-15T13:10:24.223236+0000","last_became_active":"2024-05-15T12:34:42.317054+0000","last_became_peered":"2024-05-15T12:34:42.317054+0000","last_unstale":"2024-05-15T13:10:24.223236+0000","last_undegraded":"2024-05-15T13:10:24.223236+0000","last_fullsized":"2024-05-15T13:10:24.223236+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:24.221570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:24.221570+0000","last_clean_scrub_stamp":"2024-05-15T13:10:24.221570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:42:26.838557+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":2424,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:23.033163+0000","last_change":"2024-05-15T13:10:23.034432+0000","last_active":"2024-05-15T13:10:23.033163+0000","last_peered":"2024-05-15T13:10:23.033163+0000","last_clean":"2024-05-15T13:10:23.033163+0000","last_became_active":"2024-05-15T12:34:48.742068+0000","last_became_peered":"2024-05-15T12:34:48.742068+0000","last_unstale":"2024-05-15T13:10:23.033163+0000","last_undegraded":"2024-05-15T13:10:23.033163+0000","last_fullsized":"2024-05-15T13:10:23.033163+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:23.030600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:23.030600+0000","last_clean_scrub_stamp":"2024-05-15T13:10:23.030600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:38:37.379516+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"1251'78","reported_seq":2478,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:21.334748+0000","last_change":"2024-05-15T13:10:21.336063+0000","last_active":"2024-05-15T13:10:21.334748+0000","last_peered":"2024-05-15T13:10:21.334748+0000","last_clean":"2024-05-15T13:10:21.334748+0000","last_became_active":"2024-05-15T12:32:42.889644+0000","last_became_peered":"2024-05-15T12:32:42.889644+0000","last_unstale":"2024-05-15T13:10:21.334748+0000","last_undegraded":"2024-05-15T13:10:21.334748+0000","last_fullsized":"2024-05-15T13:10:21.334748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'78","last_scrub_stamp":"2024-05-15T13:10:21.333003+0000","last_deep_scrub":"1251'78","last_deep_scrub_stamp":"2024-05-15T13:10:21.333003+0000","last_clean_scrub_stamp":"2024-05-15T13:10:21.333003+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:22:57.058726+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2428,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.704859+0000","last_change":"2024-05-15T12:32:47.722217+0000","last_active":"2024-05-15T13:10:18.704859+0000","last_peered":"2024-05-15T13:10:18.704859+0000","last_clean":"2024-05-15T13:10:18.704859+0000","last_became_active":"2024-05-15T12:32:47.716674+0000","last_became_peered":"2024-05-15T12:32:47.716674+0000","last_unstale":"2024-05-15T13:10:18.704859+0000","last_undegraded":"2024-05-15T13:10:18.704859+0000","last_fullsized":"2024-05-15T13:10:18.704859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:25:03.820946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":2425,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.721339+0000","last_change":"2024-05-15T12:34:42.947741+0000","last_active":"2024-05-15T13:10:17.721339+0000","last_peered":"2024-05-15T13:10:17.721339+0000","last_clean":"2024-05-15T13:10:17.721339+0000","last_became_active":"2024-05-15T12:34:42.942214+0000","last_became_peered":"2024-05-15T12:34:42.942214+0000","last_unstale":"2024-05-15T13:10:17.721339+0000","last_undegraded":"2024-05-15T13:10:17.721339+0000","last_fullsized":"2024-05-15T13:10:17.721339+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:08:22.005787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2429,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.678266+0000","last_change":"2024-05-15T12:32:47.459103+0000","last_active":"2024-05-15T13:10:18.678266+0000","last_peered":"2024-05-15T13:10:18.678266+0000","last_clean":"2024-05-15T13:10:18.678266+0000","last_became_active":"2024-05-15T12:32:47.430194+0000","last_became_peered":"2024-05-15T12:32:47.430194+0000","last_unstale":"2024-05-15T13:10:18.678266+0000","last_undegraded":"2024-05-15T13:10:18.678266+0000","last_fullsized":"2024-05-15T13:10:18.678266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:14:58.294854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":2416,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.726128+0000","last_change":"2024-05-15T12:34:52.363872+0000","last_active":"2024-05-15T13:10:18.726128+0000","last_peered":"2024-05-15T13:10:18.726128+0000","last_clean":"2024-05-15T13:10:18.726128+0000","last_became_active":"2024-05-15T12:34:52.357328+0000","last_became_peered":"2024-05-15T12:34:52.357328+0000","last_unstale":"2024-05-15T13:10:18.726128+0000","last_undegraded":"2024-05-15T13:10:18.726128+0000","last_fullsized":"2024-05-15T13:10:18.726128+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:05:48.456769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2416,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.648255+0000","last_change":"2024-05-15T12:32:47.463044+0000","last_active":"2024-05-15T13:10:17.648255+0000","last_peered":"2024-05-15T13:10:17.648255+0000","last_clean":"2024-05-15T13:10:17.648255+0000","last_became_active":"2024-05-15T12:32:47.432679+0000","last_became_peered":"2024-05-15T12:32:47.432679+0000","last_unstale":"2024-05-15T13:10:17.648255+0000","last_undegraded":"2024-05-15T13:10:17.648255+0000","last_fullsized":"2024-05-15T13:10:17.648255+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:13:56.912246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":2422,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.706244+0000","last_change":"2024-05-15T12:34:36.929836+0000","last_active":"2024-05-15T13:10:18.706244+0000","last_peered":"2024-05-15T13:10:18.706244+0000","last_clean":"2024-05-15T13:10:18.706244+0000","last_became_active":"2024-05-15T12:34:36.924450+0000","last_became_peered":"2024-05-15T12:34:36.924450+0000","last_unstale":"2024-05-15T13:10:18.706244+0000","last_undegraded":"2024-05-15T13:10:18.706244+0000","last_fullsized":"2024-05-15T13:10:18.706244+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:13:01.098128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2431,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.709051+0000","last_change":"2024-05-15T12:36:58.748207+0000","last_active":"2024-05-15T13:10:18.709051+0000","last_peered":"2024-05-15T13:10:18.709051+0000","last_clean":"2024-05-15T13:10:18.709051+0000","last_became_active":"2024-05-15T12:32:47.670226+0000","last_became_peered":"2024-05-15T12:32:47.670226+0000","last_unstale":"2024-05-15T13:10:18.709051+0000","last_undegraded":"2024-05-15T13:10:18.709051+0000","last_fullsized":"2024-05-15T13:10:18.709051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:41:09.623434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":2415,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.740965+0000","last_change":"2024-05-15T12:34:51.754656+0000","last_active":"2024-05-15T13:10:18.740965+0000","last_peered":"2024-05-15T13:10:18.740965+0000","last_clean":"2024-05-15T13:10:18.740965+0000","last_became_active":"2024-05-15T12:34:51.746853+0000","last_became_peered":"2024-05-15T12:34:51.746853+0000","last_unstale":"2024-05-15T13:10:18.740965+0000","last_undegraded":"2024-05-15T13:10:18.740965+0000","last_fullsized":"2024-05-15T13:10:18.740965+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:59:19.605051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2438,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.693486+0000","last_change":"2024-05-15T12:32:47.687489+0000","last_active":"2024-05-15T13:10:17.693486+0000","last_peered":"2024-05-15T13:10:17.693486+0000","last_clean":"2024-05-15T13:10:17.693486+0000","last_became_active":"2024-05-15T12:32:47.680498+0000","last_became_peered":"2024-05-15T12:32:47.680498+0000","last_unstale":"2024-05-15T13:10:17.693486+0000","last_undegraded":"2024-05-15T13:10:17.693486+0000","last_fullsized":"2024-05-15T13:10:17.693486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:09:36.650810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"1251'8","reported_seq":2441,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:24.073755+0000","last_change":"2024-05-15T13:10:24.075097+0000","last_active":"2024-05-15T13:10:24.073755+0000","last_peered":"2024-05-15T13:10:24.073755+0000","last_clean":"2024-05-15T13:10:24.073755+0000","last_became_active":"2024-05-15T12:34:50.371773+0000","last_became_peered":"2024-05-15T12:34:50.371773+0000","last_unstale":"2024-05-15T13:10:24.073755+0000","last_undegraded":"2024-05-15T13:10:24.073755+0000","last_fullsized":"2024-05-15T13:10:24.073755+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'8","last_scrub_stamp":"2024-05-15T13:10:24.072030+0000","last_deep_scrub":"1251'8","last_deep_scrub_stamp":"2024-05-15T13:10:24.072030+0000","last_clean_scrub_stamp":"2024-05-15T13:10:24.072030+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:38:41.763044+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2427,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.728099+0000","last_change":"2024-05-15T12:32:47.531833+0000","last_active":"2024-05-15T13:10:18.728099+0000","last_peered":"2024-05-15T13:10:18.728099+0000","last_clean":"2024-05-15T13:10:18.728099+0000","last_became_active":"2024-05-15T12:32:47.526275+0000","last_became_peered":"2024-05-15T12:32:47.526275+0000","last_unstale":"2024-05-15T13:10:18.728099+0000","last_undegraded":"2024-05-15T13:10:18.728099+0000","last_fullsized":"2024-05-15T13:10:18.728099+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:32:09.483290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":2419,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:23.238174+0000","last_change":"2024-05-15T13:10:23.239057+0000","last_active":"2024-05-15T13:10:23.238174+0000","last_peered":"2024-05-15T13:10:23.238174+0000","last_clean":"2024-05-15T13:10:23.238174+0000","last_became_active":"2024-05-15T12:34:38.888318+0000","last_became_peered":"2024-05-15T12:34:38.888318+0000","last_unstale":"2024-05-15T13:10:23.238174+0000","last_undegraded":"2024-05-15T13:10:23.238174+0000","last_fullsized":"2024-05-15T13:10:23.238174+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:23.235879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:23.235879+0000","last_clean_scrub_stamp":"2024-05-15T13:10:23.235879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T01:07:36.570458+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2432,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.695615+0000","last_change":"2024-05-15T12:32:48.309674+0000","last_active":"2024-05-15T13:10:18.695615+0000","last_peered":"2024-05-15T13:10:18.695615+0000","last_clean":"2024-05-15T13:10:18.695615+0000","last_became_active":"2024-05-15T12:32:48.283635+0000","last_became_peered":"2024-05-15T12:32:48.283635+0000","last_unstale":"2024-05-15T13:10:18.695615+0000","last_undegraded":"2024-05-15T13:10:18.695615+0000","last_fullsized":"2024-05-15T13:10:18.695615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:07:32.002393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":2426,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.680512+0000","last_change":"2024-05-15T12:34:48.672942+0000","last_active":"2024-05-15T13:10:17.680512+0000","last_peered":"2024-05-15T13:10:17.680512+0000","last_clean":"2024-05-15T13:10:17.680512+0000","last_became_active":"2024-05-15T12:34:48.664879+0000","last_became_peered":"2024-05-15T12:34:48.664879+0000","last_unstale":"2024-05-15T13:10:17.680512+0000","last_undegraded":"2024-05-15T13:10:17.680512+0000","last_fullsized":"2024-05-15T13:10:17.680512+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:48:54.955944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2426,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.743238+0000","last_change":"2024-05-15T12:32:48.258451+0000","last_active":"2024-05-15T13:10:18.743238+0000","last_peered":"2024-05-15T13:10:18.743238+0000","last_clean":"2024-05-15T13:10:18.743238+0000","last_became_active":"2024-05-15T12:32:48.252725+0000","last_became_peered":"2024-05-15T12:32:48.252725+0000","last_unstale":"2024-05-15T13:10:18.743238+0000","last_undegraded":"2024-05-15T13:10:18.743238+0000","last_fullsized":"2024-05-15T13:10:18.743238+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:32:45.779000+0000","last_clean_scrub_stamp":"2024-05-15T12:32:45.779000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:01:52.775805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":2422,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.724376+0000","last_change":"2024-05-15T12:34:34.676904+0000","last_active":"2024-05-15T13:10:18.724376+0000","last_peered":"2024-05-15T13:10:18.724376+0000","last_clean":"2024-05-15T13:10:18.724376+0000","last_became_active":"2024-05-15T12:34:34.660678+0000","last_became_peered":"2024-05-15T12:34:34.660678+0000","last_unstale":"2024-05-15T13:10:18.724376+0000","last_undegraded":"2024-05-15T13:10:18.724376+0000","last_fullsized":"2024-05-15T13:10:18.724376+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:39:07.124084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":2402,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.643244+0000","last_change":"2024-05-15T12:34:44.299228+0000","last_active":"2024-05-15T13:10:17.643244+0000","last_peered":"2024-05-15T13:10:17.643244+0000","last_clean":"2024-05-15T13:10:17.643244+0000","last_became_active":"2024-05-15T12:34:44.277750+0000","last_became_peered":"2024-05-15T12:34:44.277750+0000","last_unstale":"2024-05-15T13:10:17.643244+0000","last_undegraded":"2024-05-15T13:10:17.643244+0000","last_fullsized":"2024-05-15T13:10:17.643244+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:00:02.340970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":2425,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:22.008821+0000","last_change":"2024-05-15T13:10:22.009696+0000","last_active":"2024-05-15T13:10:22.008821+0000","last_peered":"2024-05-15T13:10:22.008821+0000","last_clean":"2024-05-15T13:10:22.008821+0000","last_became_active":"2024-05-15T12:34:39.369183+0000","last_became_peered":"2024-05-15T12:34:39.369183+0000","last_unstale":"2024-05-15T13:10:22.008821+0000","last_undegraded":"2024-05-15T13:10:22.008821+0000","last_fullsized":"2024-05-15T13:10:22.008821+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:22.006994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:22.006994+0000","last_clean_scrub_stamp":"2024-05-15T13:10:22.006994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:29:40.757842+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":2417,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:22.206465+0000","last_change":"2024-05-15T13:10:22.207331+0000","last_active":"2024-05-15T13:10:22.206465+0000","last_peered":"2024-05-15T13:10:22.206465+0000","last_clean":"2024-05-15T13:10:22.206465+0000","last_became_active":"2024-05-15T12:34:38.812314+0000","last_became_peered":"2024-05-15T12:34:38.812314+0000","last_unstale":"2024-05-15T13:10:22.206465+0000","last_undegraded":"2024-05-15T13:10:22.206465+0000","last_fullsized":"2024-05-15T13:10:22.206465+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:22.204826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:22.204826+0000","last_clean_scrub_stamp":"2024-05-15T13:10:22.204826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:26:09.187859+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":2401,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.673869+0000","last_change":"2024-05-15T12:34:36.784260+0000","last_active":"2024-05-15T13:10:17.673869+0000","last_peered":"2024-05-15T13:10:17.673869+0000","last_clean":"2024-05-15T13:10:17.673869+0000","last_became_active":"2024-05-15T12:34:36.777340+0000","last_became_peered":"2024-05-15T12:34:36.777340+0000","last_unstale":"2024-05-15T13:10:17.673869+0000","last_undegraded":"2024-05-15T13:10:17.673869+0000","last_fullsized":"2024-05-15T13:10:17.673869+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:27:38.620258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":2415,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.716751+0000","last_change":"2024-05-15T12:34:51.056513+0000","last_active":"2024-05-15T13:10:18.716751+0000","last_peered":"2024-05-15T13:10:18.716751+0000","last_clean":"2024-05-15T13:10:18.716751+0000","last_became_active":"2024-05-15T12:34:51.040048+0000","last_became_peered":"2024-05-15T12:34:51.040048+0000","last_unstale":"2024-05-15T13:10:18.716751+0000","last_undegraded":"2024-05-15T13:10:18.716751+0000","last_fullsized":"2024-05-15T13:10:18.716751+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":2424,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.712742+0000","last_change":"2024-05-15T12:34:51.656110+0000","last_active":"2024-05-15T13:10:17.712742+0000","last_peered":"2024-05-15T13:10:17.712742+0000","last_clean":"2024-05-15T13:10:17.712742+0000","last_became_active":"2024-05-15T12:34:51.649910+0000","last_became_peered":"2024-05-15T12:34:51.649910+0000","last_unstale":"2024-05-15T13:10:17.712742+0000","last_undegraded":"2024-05-15T13:10:17.712742+0000","last_fullsized":"2024-05-15T13:10:17.712742+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:01:47.425779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":2404,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:17.662529+0000","last_change":"2024-05-15T12:34:44.408851+0000","last_active":"2024-05-15T13:10:17.662529+0000","last_peered":"2024-05-15T13:10:17.662529+0000","last_clean":"2024-05-15T13:10:17.662529+0000","last_became_active":"2024-05-15T12:34:44.399991+0000","last_became_peered":"2024-05-15T12:34:44.399991+0000","last_unstale":"2024-05-15T13:10:17.662529+0000","last_undegraded":"2024-05-15T13:10:17.662529+0000","last_fullsized":"2024-05-15T13:10:17.662529+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:40:47.114950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":2437,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:20.323634+0000","last_change":"2024-05-15T13:10:20.324517+0000","last_active":"2024-05-15T13:10:20.323634+0000","last_peered":"2024-05-15T13:10:20.323634+0000","last_clean":"2024-05-15T13:10:20.323634+0000","last_became_active":"2024-05-15T12:34:38.549385+0000","last_became_peered":"2024-05-15T12:34:38.549385+0000","last_unstale":"2024-05-15T13:10:20.323634+0000","last_undegraded":"2024-05-15T13:10:20.323634+0000","last_fullsized":"2024-05-15T13:10:20.323634+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:20.318383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:20.318383+0000","last_clean_scrub_stamp":"2024-05-15T13:10:20.318383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:42:59.241875+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":2419,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.681880+0000","last_change":"2024-05-15T12:34:36.889179+0000","last_active":"2024-05-15T13:10:18.681880+0000","last_peered":"2024-05-15T13:10:18.681880+0000","last_clean":"2024-05-15T13:10:18.681880+0000","last_became_active":"2024-05-15T12:34:36.875596+0000","last_became_peered":"2024-05-15T12:34:36.875596+0000","last_unstale":"2024-05-15T13:10:18.681880+0000","last_undegraded":"2024-05-15T13:10:18.681880+0000","last_fullsized":"2024-05-15T13:10:18.681880+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:30:10.134778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":2415,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.695117+0000","last_change":"2024-05-15T12:34:51.659524+0000","last_active":"2024-05-15T13:10:18.695117+0000","last_peered":"2024-05-15T13:10:18.695117+0000","last_clean":"2024-05-15T13:10:18.695117+0000","last_became_active":"2024-05-15T12:34:51.653558+0000","last_became_peered":"2024-05-15T12:34:51.653558+0000","last_unstale":"2024-05-15T13:10:18.695117+0000","last_undegraded":"2024-05-15T13:10:18.695117+0000","last_fullsized":"2024-05-15T13:10:18.695117+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:43:01.117943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":2415,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.707108+0000","last_change":"2024-05-15T12:34:36.637973+0000","last_active":"2024-05-15T13:10:18.707108+0000","last_peered":"2024-05-15T13:10:18.707108+0000","last_clean":"2024-05-15T13:10:18.707108+0000","last_became_active":"2024-05-15T12:34:36.629372+0000","last_became_peered":"2024-05-15T12:34:36.629372+0000","last_unstale":"2024-05-15T13:10:18.707108+0000","last_undegraded":"2024-05-15T13:10:18.707108+0000","last_fullsized":"2024-05-15T13:10:18.707108+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:10:32.079108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":2416,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:18.714592+0000","last_change":"2024-05-15T12:34:49.358758+0000","last_active":"2024-05-15T13:10:18.714592+0000","last_peered":"2024-05-15T13:10:18.714592+0000","last_clean":"2024-05-15T13:10:18.714592+0000","last_became_active":"2024-05-15T12:34:49.324761+0000","last_became_peered":"2024-05-15T12:34:49.324761+0000","last_unstale":"2024-05-15T13:10:18.714592+0000","last_undegraded":"2024-05-15T13:10:18.714592+0000","last_fullsized":"2024-05-15T13:10:18.714592+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:29.506279+0000","last_clean_scrub_stamp":"2024-05-15T12:34:29.506279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:13:55.863763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":6703,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":13406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51,"ondisk_log_size":51,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313670,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68852,"kb_used_data":5740,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302988,"statfs":{"total":96636764160,"available":96566259712,"internally_reserved":0,"allocated":5877760,"data_stored":3157052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":7,"up_from":19,"seq":81604379075,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72576,"kb_used_data":5300,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299264,"statfs":{"total":96636764160,"available":96562446336,"internally_reserved":0,"allocated":5427200,"data_stored":2757884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":19,"seq":81604379081,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72988,"kb_used_data":5712,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298852,"statfs":{"total":96636764160,"available":96562024448,"internally_reserved":0,"allocated":5849088,"data_stored":3156580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444487,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76740,"kb_used_data":5308,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295100,"statfs":{"total":96636764160,"available":96558182400,"internally_reserved":0,"allocated":5435392,"data_stored":2757816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444485,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72544,"kb_used_data":5272,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299296,"statfs":{"total":96636764160,"available":96562479104,"internally_reserved":0,"allocated":5398528,"data_stored":2756889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542601,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72604,"kb_used_data":5328,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299236,"statfs":{"total":96636764160,"available":96562417664,"internally_reserved":0,"allocated":5455872,"data_stored":2759133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68420,"kb_used_data":5308,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303420,"statfs":{"total":96636764160,"available":96566702080,"internally_reserved":0,"allocated":5435392,"data_stored":2758174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575303,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72580,"kb_used_data":5304,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299260,"statfs":{"total":96636764160,"available":96562442240,"internally_reserved":0,"allocated":5431296,"data_stored":2757936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:10:26.824 INFO:tasks.ceph:pgid 64.1f last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.824 INFO:tasks.ceph:pgid 64.1e last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.824 INFO:tasks.ceph:pgid 64.1d last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.824 INFO:tasks.ceph:pgid 64.1c last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.824 INFO:tasks.ceph:pgid 64.1b last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.824 INFO:tasks.ceph:pgid 64.1a last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.824 INFO:tasks.ceph:pgid 64.19 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.825 INFO:tasks.ceph:pgid 64.18 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.825 INFO:tasks.ceph:pgid 64.17 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.825 INFO:tasks.ceph:pgid 64.16 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.825 INFO:tasks.ceph:pgid 64.15 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.825 INFO:tasks.ceph:pgid 64.14 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.825 INFO:tasks.ceph:pgid 64.13 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.825 INFO:tasks.ceph:pgid 64.12 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.825 INFO:tasks.ceph:pgid 64.11 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.825 INFO:tasks.ceph:pgid 64.10 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.825 INFO:tasks.ceph:pgid 64.f last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.826 INFO:tasks.ceph:pgid 64.e last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.826 INFO:tasks.ceph:pgid 64.d last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.826 INFO:tasks.ceph:pgid 64.c last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.826 INFO:tasks.ceph:pgid 64.b last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.826 INFO:tasks.ceph:pgid 64.a last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.826 INFO:tasks.ceph:pgid 64.9 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.826 INFO:tasks.ceph:pgid 64.8 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.826 INFO:tasks.ceph:pgid 64.7 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.826 INFO:tasks.ceph:pgid 64.5 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.827 INFO:tasks.ceph:pgid 64.3 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.827 INFO:tasks.ceph:pgid 64.1 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.827 INFO:tasks.ceph:pgid 64.0 last_scrub_stamp 2024-05-15T12:35:14.324269+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=35, tm_sec=14, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.827 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.827 INFO:tasks.ceph:pgid 7.16 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.827 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.827 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.827 INFO:tasks.ceph:pgid 7.b last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.827 INFO:tasks.ceph:pgid 7.a last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.828 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.828 INFO:tasks.ceph:pgid 7.8 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.828 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.828 INFO:tasks.ceph:pgid 7.e last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.828 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-15T12:32:45.779000+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=32, tm_sec=45, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.828 INFO:tasks.ceph:pgid 7.6 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.828 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-15T12:32:45.779000+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=32, tm_sec=45, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.828 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.828 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-05-15T12:32:45.779000+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=32, tm_sec=45, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.829 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.829 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-15T12:32:45.779000+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=32, tm_sec=45, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.829 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.829 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-15T12:32:45.779000+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=32, tm_sec=45, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.829 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-05-15T12:32:45.779000+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=32, tm_sec=45, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.829 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-15T12:32:45.779000+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=32, tm_sec=45, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.829 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.829 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-15T12:32:45.779000+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=32, tm_sec=45, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.829 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.829 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.830 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.830 INFO:tasks.ceph:pgid 7.1c last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.830 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.830 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.830 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.830 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.830 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.830 INFO:tasks.ceph:pgid 7.1b last_scrub_stamp 2024-05-15T12:34:29.506279+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=12, tm_min=34, tm_sec=29, tm_wday=2, tm_yday=136, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=15, tm_hour=13, tm_min=10, tm_sec=18, tm_wday=2, tm_yday=136, tm_isdst=0) 2024-05-15T13:10:26.830 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-15T13:10:32.746 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:01.954 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:32.748 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:01.957 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:32.766 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:01.975 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:32.796 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:02.004 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:32.797 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:01.964 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:32.798 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:01.966 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:32.800 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:02.009 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:32.813 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:01.980 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:32.820 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:02.029 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:42.752 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:11.960 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:42.754 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:11.963 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:42.769 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:11.977 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:42.800 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:11.967 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:42.802 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:11.969 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:42.806 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:12.015 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:42.808 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:12.016 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:42.817 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:11.983 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:42.822 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:12.030 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:46.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-15T13:10:46.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.952+0000 7f6fcbf98640 10 monclient: get_monmap_and_config 2024-05-15T13:10:46.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.952+0000 7f6fcbf98640 10 monclient: build_initial_monmap 2024-05-15T13:10:46.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.952+0000 7f6fcbf98640 10 monclient: monmap: 2024-05-15T13:10:46.943 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:46.943 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:46.943 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:46.953695+0000 2024-05-15T13:10:46.943 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:46.953695+0000 2024-05-15T13:10:46.943 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:46.944 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:46.944 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:46.944 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:46.944 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:46.944 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:46.944 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:46.944 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:46.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.952+0000 7f6fcbf98640 1 Processor -- start 2024-05-15T13:10:46.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 1 -- start start 2024-05-15T13:10:46.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 10 monclient: init 2024-05-15T13:10:46.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fc40b5410 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fc41400a0 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6fc4146310 0x7f6fc40a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 10 monclient(hunting): picked mon.noname-a con 0x7f6fc4146310 addr v2:172.21.15.49:3300/0 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc40ab470 con 0x7f6fc41400a0 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc40aa0c0 con 0x7f6fc40b5410 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc40b57f0 con 0x7f6fc4146310 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.953+0000 7f6fcbf98640 10 monclient(hunting): authenticate will time out at 3805.116276s 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.954+0000 7f6fc9d34640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6fc40b5410 0x7f6fc413f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.49:57150/0 (socket says 172.21.15.49:57150) 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.954+0000 7f6fc9d34640 1 -- 172.21.15.49:0/1362709083 learned_addr learned my addr 172.21.15.49:0/1362709083 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:46.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.954+0000 7f6fca535640 1 --2- 172.21.15.49:0/1362709083 >> v2:172.21.15.49:3300/0 conn(0x7f6fc4146310 0x7f6fc40a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:46.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.955+0000 7f6fca535640 10 monclient(hunting): get_auth_request con 0x7f6fc4146310 auth_method 0 2024-05-15T13:10:46.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.955+0000 7f6fca535640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:46.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.955+0000 7f6fca535640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:46.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.955+0000 7f6fca535640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:46.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.956+0000 7f6fca535640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:46.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.956+0000 7f6fca535640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:46.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.956+0000 7f6fca535640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:46.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.957+0000 7f6fc8d32640 1 -- 172.21.15.49:0/1362709083 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275249068 0 0) 0x7f6fc40aa0c0 con 0x7f6fc40b5410 2024-05-15T13:10:46.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.957+0000 7f6fc8d32640 10 monclient(hunting): _negotiate 2024-05-15T13:10:46.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.957+0000 7f6fc8d32640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:46.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.957+0000 7f6fc8d32640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:46.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.957+0000 7f6fc8d32640 10 monclient(hunting): my global_id is 22105 2024-05-15T13:10:46.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.957+0000 7f6fc8d32640 1 -- 172.21.15.49:0/1362709083 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fac003520 con 0x7f6fc40b5410 2024-05-15T13:10:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.959+0000 7f6fca535640 10 monclient(hunting): handle_auth_done global_id 16245 payload 274 2024-05-15T13:10:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.959+0000 7f6fca535640 1 -- 172.21.15.49:0/1362709083 >> v1:172.21.15.88:6789/0 conn(0x7f6fc40b5410 legacy=0x7f6fc413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.959+0000 7f6fca535640 1 -- 172.21.15.49:0/1362709083 >> v1:172.21.15.49:6789/0 conn(0x7f6fc41400a0 legacy=0x7f6fc4145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.959+0000 7f6fca535640 10 monclient: _finish_hunting 0 2024-05-15T13:10:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.959+0000 7f6fca535640 1 monclient: found mon.noname-a 2024-05-15T13:10:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.959+0000 7f6fca535640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.959+0000 7f6fca535640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T13:10:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.959+0000 7f6fca535640 1 -- 172.21.15.49:0/1362709083 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc40aa5e0 con 0x7f6fc4146310 2024-05-15T13:10:46.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.962+0000 7f6fca535640 1 --2- 172.21.15.49:0/1362709083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 0x7f6fc40a9960 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0097a0 tx=0x7f6fbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459d776c33a21739 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:46.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 1 -- 172.21.15.49:0/1362709083 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc035020 con 0x7f6fc4146310 2024-05-15T13:10:46.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:46.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:10:46.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 10 monclient: dump: 2024-05-15T13:10:46.958 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:46.958 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:46.958 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:46.958 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:46.958 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 10 monclient: _finish_auth 0 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 10 monclient: _check_auth_tickets 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 1 -- 172.21.15.49:0/1362709083 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc031830 con 0x7f6fc4146310 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 1 -- 172.21.15.49:0/1362709083 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc031c00 con 0x7f6fc4146310 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 5 monclient: authenticate success, global_id 16245 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 4 set_mon_vals no callback set 2024-05-15T13:10:46.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fc8d32640 10 monclient: dump: 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:46.960 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:46.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 10 monclient: shutdown 2024-05-15T13:10:46.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:46.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 1 -- 172.21.15.49:0/1362709083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 msgr2=0x7f6fc40a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:46.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 1 --2- 172.21.15.49:0/1362709083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 0x7f6fc40a9960 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0097a0 tx=0x7f6fbc02b8d0 comp rx=0 tx=0).stop 2024-05-15T13:10:46.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 1 -- 172.21.15.49:0/1362709083 shutdown_connections 2024-05-15T13:10:46.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 1 --2- 172.21.15.49:0/1362709083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 0x7f6fc40a9960 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:46.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 1 -- 172.21.15.49:0/1362709083 >> 172.21.15.49:0/1362709083 conn(0x7f6fc4066f00 msgr2=0x7f6fc4067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:46.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 1 -- 172.21.15.49:0/1362709083 shutdown_connections 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.967+0000 7f6fcbf98640 1 -- 172.21.15.49:0/1362709083 wait complete. 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.968+0000 7f6fcbf98640 10 monclient: build_initial_monmap 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.968+0000 7f6fcbf98640 10 monclient: monmap: 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.968+0000 7f6fcbf98640 1 librados: starting msgr at 2024-05-15T13:10:46.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.968+0000 7f6fcbf98640 1 librados: starting objecter 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.968+0000 7f6fcbf98640 1 Processor -- start 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.968+0000 7f6fcbf98640 1 -- start start 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.968+0000 7f6fcbf98640 1 librados: setting wanted keys 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.968+0000 7f6fcbf98640 1 librados: calling monclient init 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.968+0000 7f6fcbf98640 10 monclient: init 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fc40b5410 0x7f6fc4141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 10 monclient(hunting): picked mon.noname-c con 0x7f6fc40b5410 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc41400a0 0x7f6fc413d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 10 monclient(hunting): picked mon.noname-b con 0x7f6fc41400a0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 0x7f6fc413dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:46.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 10 monclient(hunting): picked mon.noname-a con 0x7f6fc4146310 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:46.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:46.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc414df40 con 0x7f6fc4146310 2024-05-15T13:10:46.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:46.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc40a9ea0 con 0x7f6fc41400a0 2024-05-15T13:10:46.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:46.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc414e0c0 con 0x7f6fc40b5410 2024-05-15T13:10:46.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:46.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.969+0000 7f6fcbf98640 10 monclient(hunting): authenticate will time out at 3805.131939s 2024-05-15T13:10:46.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fca535640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 0x7f6fc413dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:46.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fc9533640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc41400a0 0x7f6fc413d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:46.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fc9d34640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fc40b5410 0x7f6fc4141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:46.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fca535640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 0x7f6fc413dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56054/0 (socket says 172.21.15.49:56054) 2024-05-15T13:10:46.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fca535640 1 -- 172.21.15.49:0/2972618690 learned_addr learned my addr 172.21.15.49:0/2972618690 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:46.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fca535640 10 monclient(hunting): get_auth_request con 0x7f6fc4146310 auth_method 0 2024-05-15T13:10:46.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fca535640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:46.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fca535640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:46.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fca535640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:46.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fc9533640 10 monclient(hunting): get_auth_request con 0x7f6fc41400a0 auth_method 0 2024-05-15T13:10:46.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fc9533640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:46.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fc9533640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:46.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fc9533640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:46.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fc9d34640 10 monclient(hunting): get_auth_request con 0x7f6fc40b5410 auth_method 0 2024-05-15T13:10:46.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fc9d34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:46.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fc9d34640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:46.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fc9d34640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:46.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fca535640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:46.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fca535640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:46.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.970+0000 7f6fca535640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:46.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.971+0000 7f6fc9533640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:46.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.971+0000 7f6fc9533640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:46.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.971+0000 7f6fc9533640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:46.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.971+0000 7f6fc9d34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:46.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.971+0000 7f6fc9d34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:46.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.971+0000 7f6fc9d34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:46.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.973+0000 7f6fca535640 10 monclient(hunting): handle_auth_done global_id 16251 payload 824 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.973+0000 7f6fca535640 1 -- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fc40b5410 msgr2=0x7f6fc4141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.973+0000 7f6fca535640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fc40b5410 0x7f6fc4141c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.973+0000 7f6fca535640 1 -- 172.21.15.49:0/2972618690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc41400a0 msgr2=0x7f6fc413d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.973+0000 7f6fca535640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc41400a0 0x7f6fc413d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.973+0000 7f6fca535640 10 monclient: _finish_hunting 0 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.973+0000 7f6fca535640 1 monclient: found mon.noname-a 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.973+0000 7f6fca535640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.973+0000 7f6fca535640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.49:3300/0 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.973+0000 7f6fca535640 1 -- 172.21.15.49:0/2972618690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4141200 con 0x7f6fc4146310 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.976+0000 7f6fca535640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 0x7f6fc413dee0 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc034040 tx=0x7f6fbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.976+0000 7f6fc27fc640 10 client.?.objecter ms_handle_connect 0x7f6fc4146310 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.976+0000 7f6fc27fc640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 1 -- 172.21.15.49:0/2972618690 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc040070 con 0x7f6fc4146310 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:46.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 10 monclient: dump: 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 10 monclient: _finish_auth 0 2024-05-15T13:10:46.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 10 monclient: _check_auth_tickets 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 1 -- 172.21.15.49:0/2972618690 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc03a910 con 0x7f6fc4146310 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 1 -- 172.21.15.49:0/2972618690 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc035040 con 0x7f6fc4146310 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fcbf98640 5 monclient: authenticate success, global_id 16251 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fcbf98640 10 monclient: _renew_subs 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fcbf98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fcbf98640 1 -- 172.21.15.49:0/2972618690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc41414e0 con 0x7f6fc4146310 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc27fc640 10 monclient: dump: 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fc3fff640 4 set_mon_vals no callback set 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fcbf98640 10 client.16251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fcbf98640 10 monclient: _renew_subs 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fcbf98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fcbf98640 1 -- 172.21.15.49:0/2972618690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc413e600 con 0x7f6fc4146310 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.980+0000 7f6fcbf98640 1 librados: init done 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.981+0000 7f6fcbf98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.981+0000 7f6fcbf98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:46.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.981+0000 7f6fcbf98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.49:3300/0 2024-05-15T13:10:46.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:46.981+0000 7f6fcbf98640 1 -- 172.21.15.49:0/2972618690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc404f0e0 con 0x7f6fc4146310 2024-05-15T13:10:46.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.000+0000 7f6fc27fc640 1 -- 172.21.15.49:0/2972618690 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6fbc04b060 con 0x7f6fc4146310 2024-05-15T13:10:46.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.001+0000 7f6fc27fc640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f6fb006ceb0 0x7f6fb006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:46.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.001+0000 7f6fc9d34640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f6fb006ceb0 0x7f6fb006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:46.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.001+0000 7f6fc9d34640 10 monclient: get_auth_request con 0x7f6fb006ceb0 auth_method 0 2024-05-15T13:10:46.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.001+0000 7f6fc9d34640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f6fb006ceb0 0x7f6fb006f370 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4001560 tx=0x7f6fb4001a90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:46.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.001+0000 7f6fc27fc640 10 client.16251.objecter ms_handle_connect 0x7f6fb006ceb0 2024-05-15T13:10:47.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.261+0000 7f6fc27fc640 1 -- 172.21.15.49:0/2972618690 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f6fbc0783b0 con 0x7f6fc4146310 2024-05-15T13:10:47.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.261+0000 7f6fc27fc640 10 client.16251.objecter ms_dispatch 0x7f6fc4000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:47.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.261+0000 7f6fc27fc640 3 client.16251.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:47.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.262+0000 7f6fc27fc640 3 client.16251.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:47.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.262+0000 7f6fc27fc640 20 client.16251.objecter dump_active .. 0 homeless 2024-05-15T13:10:47.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.262+0000 7f6fc27fc640 1 -- 172.21.15.49:0/2972618690 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6fbc07a310 con 0x7f6fc4146310 2024-05-15T13:10:47.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.262+0000 7f6fc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:47.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.262+0000 7f6fc27fc640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:47.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.447+0000 7f6fcbf98640 1 -- 172.21.15.49:0/2972618690 --> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fc4146ad0 con 0x7f6fb006ceb0 2024-05-15T13:10:47.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.453+0000 7f6fc27fc640 1 -- 172.21.15.49:0/2972618690 <== mgr.4110 v2:172.21.15.49:6800/3444323000 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+197205 (secure 0 0 0) 0x7f6fc4146ad0 con 0x7f6fb006ceb0 2024-05-15T13:10:47.447 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-05-15T13:10:47.447 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-05-15T13:10:47.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 10 librados: watch_flush enter 2024-05-15T13:10:47.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 10 librados: watch_flush exit 2024-05-15T13:10:47.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 20 client.16251.objecter shutdown clearing up homeless session... 2024-05-15T13:10:47.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 10 client.16251.objecter successfully canceled tick 2024-05-15T13:10:47.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 -- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f6fb006ceb0 msgr2=0x7f6fb006f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:47.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f6fb006ceb0 0x7f6fb006f370 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4001560 tx=0x7f6fb4001a90 comp rx=0 tx=0).stop 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 10 monclient: shutdown 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 -- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 msgr2=0x7f6fc413dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 0x7f6fc413dee0 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc034040 tx=0x7f6fbc0046c0 comp rx=0 tx=0).stop 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 -- 172.21.15.49:0/2972618690 shutdown_connections 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fc4146310 0x7f6fc413dee0 unknown :-1 s=CLOSED pgs=3274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f6fb006ceb0 0x7f6fb006f370 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc41400a0 0x7f6fc413d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 --2- 172.21.15.49:0/2972618690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fc40b5410 0x7f6fc4141c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 -- 172.21.15.49:0/2972618690 >> 172.21.15.49:0/2972618690 conn(0x7f6fc4066f00 msgr2=0x7f6fc40b73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 -- 172.21.15.49:0/2972618690 shutdown_connections 2024-05-15T13:10:47.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 -- 172.21.15.49:0/2972618690 wait complete. 2024-05-15T13:10:47.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.456+0000 7f6fcbf98640 1 librados: shutdown 2024-05-15T13:10:47.458 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2404,"stamp":"2024-05-15T13:10:47.057835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465988,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":170,"num_write_kb":1429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":581448,"kb_used_data":43272,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754393272,"statfs":{"total":773094113280,"available":772498710528,"internally_reserved":0,"allocated":44310528,"data_stored":22861464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002514"},"pg_stats":[{"pgid":"64.1f","version":"1251'6","reported_seq":2392,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:38.469132+0000","last_change":"2024-05-15T13:10:38.470058+0000","last_active":"2024-05-15T13:10:38.469132+0000","last_peered":"2024-05-15T13:10:38.469132+0000","last_clean":"2024-05-15T13:10:38.469132+0000","last_became_active":"2024-05-15T12:35:49.922024+0000","last_became_peered":"2024-05-15T12:35:49.922024+0000","last_unstale":"2024-05-15T13:10:38.469132+0000","last_undegraded":"2024-05-15T13:10:38.469132+0000","last_fullsized":"2024-05-15T13:10:38.469132+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'6","last_scrub_stamp":"2024-05-15T13:10:38.467562+0000","last_deep_scrub":"1251'6","last_deep_scrub_stamp":"2024-05-15T13:10:38.467562+0000","last_clean_scrub_stamp":"2024-05-15T13:10:38.467562+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:46:15.165388+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1e","version":"0'0","reported_seq":2383,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:34.072572+0000","last_change":"2024-05-15T13:10:34.073353+0000","last_active":"2024-05-15T13:10:34.072572+0000","last_peered":"2024-05-15T13:10:34.072572+0000","last_clean":"2024-05-15T13:10:34.072572+0000","last_became_active":"2024-05-15T12:35:32.314677+0000","last_became_peered":"2024-05-15T12:35:32.314677+0000","last_unstale":"2024-05-15T13:10:34.072572+0000","last_undegraded":"2024-05-15T13:10:34.072572+0000","last_fullsized":"2024-05-15T13:10:34.072572+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:34.070957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:34.070957+0000","last_clean_scrub_stamp":"2024-05-15T13:10:34.070957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:29:51.139962+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1d","version":"1251'4","reported_seq":2374,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:31.307120+0000","last_change":"2024-05-15T13:10:31.307120+0000","last_active":"2024-05-15T13:10:31.307120+0000","last_peered":"2024-05-15T13:10:31.307120+0000","last_clean":"2024-05-15T13:10:31.307120+0000","last_became_active":"2024-05-15T12:36:16.182561+0000","last_became_peered":"2024-05-15T12:36:16.182561+0000","last_unstale":"2024-05-15T13:10:31.307120+0000","last_undegraded":"2024-05-15T13:10:31.307120+0000","last_fullsized":"2024-05-15T13:10:31.307120+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'4","last_scrub_stamp":"2024-05-15T13:10:31.302448+0000","last_deep_scrub":"1251'4","last_deep_scrub_stamp":"2024-05-15T13:10:31.302448+0000","last_clean_scrub_stamp":"2024-05-15T13:10:31.302448+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:01:32.357144+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1c","version":"0'0","reported_seq":2385,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:32.353172+0000","last_change":"2024-05-15T13:10:32.353172+0000","last_active":"2024-05-15T13:10:32.353172+0000","last_peered":"2024-05-15T13:10:32.353172+0000","last_clean":"2024-05-15T13:10:32.353172+0000","last_became_active":"2024-05-15T12:35:47.561938+0000","last_became_peered":"2024-05-15T12:35:47.561938+0000","last_unstale":"2024-05-15T13:10:32.353172+0000","last_undegraded":"2024-05-15T13:10:32.353172+0000","last_fullsized":"2024-05-15T13:10:32.353172+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:32.350865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:32.350865+0000","last_clean_scrub_stamp":"2024-05-15T13:10:32.350865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:38:34.582879+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1b","version":"1251'2","reported_seq":2394,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:31.346695+0000","last_change":"2024-05-15T13:10:31.347704+0000","last_active":"2024-05-15T13:10:31.346695+0000","last_peered":"2024-05-15T13:10:31.346695+0000","last_clean":"2024-05-15T13:10:31.346695+0000","last_became_active":"2024-05-15T12:35:31.337467+0000","last_became_peered":"2024-05-15T12:35:31.337467+0000","last_unstale":"2024-05-15T13:10:31.346695+0000","last_undegraded":"2024-05-15T13:10:31.346695+0000","last_fullsized":"2024-05-15T13:10:31.346695+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:31.344972+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:31.344972+0000","last_clean_scrub_stamp":"2024-05-15T13:10:31.344972+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:57:08.214853+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.1a","version":"1251'8","reported_seq":2378,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:27.340433+0000","last_change":"2024-05-15T13:10:27.341507+0000","last_active":"2024-05-15T13:10:27.340433+0000","last_peered":"2024-05-15T13:10:27.340433+0000","last_clean":"2024-05-15T13:10:27.340433+0000","last_became_active":"2024-05-15T12:35:44.321742+0000","last_became_peered":"2024-05-15T12:35:44.321742+0000","last_unstale":"2024-05-15T13:10:27.340433+0000","last_undegraded":"2024-05-15T13:10:27.340433+0000","last_fullsized":"2024-05-15T13:10:27.340433+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'8","last_scrub_stamp":"2024-05-15T13:10:27.338755+0000","last_deep_scrub":"1251'8","last_deep_scrub_stamp":"2024-05-15T13:10:27.338755+0000","last_clean_scrub_stamp":"2024-05-15T13:10:27.338755+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:58:27.626597+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.19","version":"1251'2","reported_seq":2387,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:36.399274+0000","last_change":"2024-05-15T13:10:36.400165+0000","last_active":"2024-05-15T13:10:36.399274+0000","last_peered":"2024-05-15T13:10:36.399274+0000","last_clean":"2024-05-15T13:10:36.399274+0000","last_became_active":"2024-05-15T12:35:49.195589+0000","last_became_peered":"2024-05-15T12:35:49.195589+0000","last_unstale":"2024-05-15T13:10:36.399274+0000","last_undegraded":"2024-05-15T13:10:36.399274+0000","last_fullsized":"2024-05-15T13:10:36.399274+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:36.397790+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:36.397790+0000","last_clean_scrub_stamp":"2024-05-15T13:10:36.397790+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:50:39.202518+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.18","version":"1251'6","reported_seq":2376,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:28.273287+0000","last_change":"2024-05-15T13:10:28.274331+0000","last_active":"2024-05-15T13:10:28.273287+0000","last_peered":"2024-05-15T13:10:28.273287+0000","last_clean":"2024-05-15T13:10:28.273287+0000","last_became_active":"2024-05-15T12:35:44.403250+0000","last_became_peered":"2024-05-15T12:35:44.403250+0000","last_unstale":"2024-05-15T13:10:28.273287+0000","last_undegraded":"2024-05-15T13:10:28.273287+0000","last_fullsized":"2024-05-15T13:10:28.273287+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'6","last_scrub_stamp":"2024-05-15T13:10:28.271567+0000","last_deep_scrub":"1251'6","last_deep_scrub_stamp":"2024-05-15T13:10:28.271567+0000","last_clean_scrub_stamp":"2024-05-15T13:10:28.271567+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:17:42.515602+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.17","version":"1251'4","reported_seq":2382,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:37.725422+0000","last_change":"2024-05-15T13:10:37.726512+0000","last_active":"2024-05-15T13:10:37.725422+0000","last_peered":"2024-05-15T13:10:37.725422+0000","last_clean":"2024-05-15T13:10:37.725422+0000","last_became_active":"2024-05-15T12:35:50.518230+0000","last_became_peered":"2024-05-15T12:35:50.518230+0000","last_unstale":"2024-05-15T13:10:37.725422+0000","last_undegraded":"2024-05-15T13:10:37.725422+0000","last_fullsized":"2024-05-15T13:10:37.725422+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'4","last_scrub_stamp":"2024-05-15T13:10:37.723814+0000","last_deep_scrub":"1251'4","last_deep_scrub_stamp":"2024-05-15T13:10:37.723814+0000","last_clean_scrub_stamp":"2024-05-15T13:10:37.723814+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:40:39.853613+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.16","version":"1251'8","reported_seq":2372,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:37.268448+0000","last_change":"2024-05-15T13:10:37.270081+0000","last_active":"2024-05-15T13:10:37.268448+0000","last_peered":"2024-05-15T13:10:37.268448+0000","last_clean":"2024-05-15T13:10:37.268448+0000","last_became_active":"2024-05-15T12:35:47.446210+0000","last_became_peered":"2024-05-15T12:35:47.446210+0000","last_unstale":"2024-05-15T13:10:37.268448+0000","last_undegraded":"2024-05-15T13:10:37.268448+0000","last_fullsized":"2024-05-15T13:10:37.268448+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'8","last_scrub_stamp":"2024-05-15T13:10:37.265879+0000","last_deep_scrub":"1251'8","last_deep_scrub_stamp":"2024-05-15T13:10:37.265879+0000","last_clean_scrub_stamp":"2024-05-15T13:10:37.265879+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:53:07.420392+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.15","version":"1251'4","reported_seq":2389,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:37.396988+0000","last_change":"2024-05-15T13:10:37.397900+0000","last_active":"2024-05-15T13:10:37.396988+0000","last_peered":"2024-05-15T13:10:37.396988+0000","last_clean":"2024-05-15T13:10:37.396988+0000","last_became_active":"2024-05-15T12:35:49.823395+0000","last_became_peered":"2024-05-15T12:35:49.823395+0000","last_unstale":"2024-05-15T13:10:37.396988+0000","last_undegraded":"2024-05-15T13:10:37.396988+0000","last_fullsized":"2024-05-15T13:10:37.396988+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'4","last_scrub_stamp":"2024-05-15T13:10:37.395439+0000","last_deep_scrub":"1251'4","last_deep_scrub_stamp":"2024-05-15T13:10:37.395439+0000","last_clean_scrub_stamp":"2024-05-15T13:10:37.395439+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:45:13.896544+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.14","version":"0'0","reported_seq":2379,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:32.255231+0000","last_change":"2024-05-15T13:10:32.255231+0000","last_active":"2024-05-15T13:10:32.255231+0000","last_peered":"2024-05-15T13:10:32.255231+0000","last_clean":"2024-05-15T13:10:32.255231+0000","last_became_active":"2024-05-15T12:35:47.926796+0000","last_became_peered":"2024-05-15T12:35:47.926796+0000","last_unstale":"2024-05-15T13:10:32.255231+0000","last_undegraded":"2024-05-15T13:10:32.255231+0000","last_fullsized":"2024-05-15T13:10:32.255231+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:32.252975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:32.252975+0000","last_clean_scrub_stamp":"2024-05-15T13:10:32.252975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:42:10.853311+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.13","version":"1251'2","reported_seq":2395,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:30.310453+0000","last_change":"2024-05-15T13:10:30.310453+0000","last_active":"2024-05-15T13:10:30.310453+0000","last_peered":"2024-05-15T13:10:30.310453+0000","last_clean":"2024-05-15T13:10:30.310453+0000","last_became_active":"2024-05-15T12:35:28.182751+0000","last_became_peered":"2024-05-15T12:35:28.182751+0000","last_unstale":"2024-05-15T13:10:30.310453+0000","last_undegraded":"2024-05-15T13:10:30.310453+0000","last_fullsized":"2024-05-15T13:10:30.310453+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:30.307510+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:30.307510+0000","last_clean_scrub_stamp":"2024-05-15T13:10:30.307510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:28:14.862918+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.12","version":"1251'2","reported_seq":2393,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:33.603382+0000","last_change":"2024-05-15T13:10:33.604398+0000","last_active":"2024-05-15T13:10:33.603382+0000","last_peered":"2024-05-15T13:10:33.603382+0000","last_clean":"2024-05-15T13:10:33.603382+0000","last_became_active":"2024-05-15T12:35:27.719548+0000","last_became_peered":"2024-05-15T12:35:27.719548+0000","last_unstale":"2024-05-15T13:10:33.603382+0000","last_undegraded":"2024-05-15T13:10:33.603382+0000","last_fullsized":"2024-05-15T13:10:33.603382+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:33.601721+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:33.601721+0000","last_clean_scrub_stamp":"2024-05-15T13:10:33.601721+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:35:28.523562+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.11","version":"1251'2","reported_seq":2383,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:36.629399+0000","last_change":"2024-05-15T13:10:36.630383+0000","last_active":"2024-05-15T13:10:36.629399+0000","last_peered":"2024-05-15T13:10:36.629399+0000","last_clean":"2024-05-15T13:10:36.629399+0000","last_became_active":"2024-05-15T12:35:50.426351+0000","last_became_peered":"2024-05-15T12:35:50.426351+0000","last_unstale":"2024-05-15T13:10:36.629399+0000","last_undegraded":"2024-05-15T13:10:36.629399+0000","last_fullsized":"2024-05-15T13:10:36.629399+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:36.627775+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:36.627775+0000","last_clean_scrub_stamp":"2024-05-15T13:10:36.627775+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:18:56.777227+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.10","version":"1251'6","reported_seq":2381,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:35.144206+0000","last_change":"2024-05-15T13:10:35.145135+0000","last_active":"2024-05-15T13:10:35.144206+0000","last_peered":"2024-05-15T13:10:35.144206+0000","last_clean":"2024-05-15T13:10:35.144206+0000","last_became_active":"2024-05-15T12:35:34.587282+0000","last_became_peered":"2024-05-15T12:35:34.587282+0000","last_unstale":"2024-05-15T13:10:35.144206+0000","last_undegraded":"2024-05-15T13:10:35.144206+0000","last_fullsized":"2024-05-15T13:10:35.144206+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'6","last_scrub_stamp":"2024-05-15T13:10:35.142589+0000","last_deep_scrub":"1251'6","last_deep_scrub_stamp":"2024-05-15T13:10:35.142589+0000","last_clean_scrub_stamp":"2024-05-15T13:10:35.142589+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:46:50.440758+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.f","version":"1251'4","reported_seq":2374,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:29.322907+0000","last_change":"2024-05-15T13:10:29.323946+0000","last_active":"2024-05-15T13:10:29.322907+0000","last_peered":"2024-05-15T13:10:29.322907+0000","last_clean":"2024-05-15T13:10:29.322907+0000","last_became_active":"2024-05-15T12:35:47.063782+0000","last_became_peered":"2024-05-15T12:35:47.063782+0000","last_unstale":"2024-05-15T13:10:29.322907+0000","last_undegraded":"2024-05-15T13:10:29.322907+0000","last_fullsized":"2024-05-15T13:10:29.322907+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'4","last_scrub_stamp":"2024-05-15T13:10:29.321288+0000","last_deep_scrub":"1251'4","last_deep_scrub_stamp":"2024-05-15T13:10:29.321288+0000","last_clean_scrub_stamp":"2024-05-15T13:10:29.321288+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:39:33.771274+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.e","version":"1251'2","reported_seq":2392,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:35.631663+0000","last_change":"2024-05-15T13:10:35.632712+0000","last_active":"2024-05-15T13:10:35.631663+0000","last_peered":"2024-05-15T13:10:35.631663+0000","last_clean":"2024-05-15T13:10:35.631663+0000","last_became_active":"2024-05-15T12:35:28.033694+0000","last_became_peered":"2024-05-15T12:35:28.033694+0000","last_unstale":"2024-05-15T13:10:35.631663+0000","last_undegraded":"2024-05-15T13:10:35.631663+0000","last_fullsized":"2024-05-15T13:10:35.631663+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:35.630002+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:35.630002+0000","last_clean_scrub_stamp":"2024-05-15T13:10:35.630002+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:20:38.571595+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.d","version":"1251'6","reported_seq":2385,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:38.766656+0000","last_change":"2024-05-15T13:10:38.767681+0000","last_active":"2024-05-15T13:10:38.766656+0000","last_peered":"2024-05-15T13:10:38.766656+0000","last_clean":"2024-05-15T13:10:38.766656+0000","last_became_active":"2024-05-15T12:35:50.586234+0000","last_became_peered":"2024-05-15T12:35:50.586234+0000","last_unstale":"2024-05-15T13:10:38.766656+0000","last_undegraded":"2024-05-15T13:10:38.766656+0000","last_fullsized":"2024-05-15T13:10:38.766656+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'6","last_scrub_stamp":"2024-05-15T13:10:38.765055+0000","last_deep_scrub":"1251'6","last_deep_scrub_stamp":"2024-05-15T13:10:38.765055+0000","last_clean_scrub_stamp":"2024-05-15T13:10:38.765055+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:07:36.365109+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.c","version":"1251'2","reported_seq":2384,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:26.043204+0000","last_change":"2024-05-15T13:10:26.044193+0000","last_active":"2024-05-15T13:10:26.043204+0000","last_peered":"2024-05-15T13:10:26.043204+0000","last_clean":"2024-05-15T13:10:26.043204+0000","last_became_active":"2024-05-15T12:35:39.550911+0000","last_became_peered":"2024-05-15T12:35:39.550911+0000","last_unstale":"2024-05-15T13:10:26.043204+0000","last_undegraded":"2024-05-15T13:10:26.043204+0000","last_fullsized":"2024-05-15T13:10:26.043204+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:26.041547+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:26.041547+0000","last_clean_scrub_stamp":"2024-05-15T13:10:26.041547+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:06:50.373170+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.b","version":"1251'4","reported_seq":2384,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:33.113687+0000","last_change":"2024-05-15T13:10:33.114712+0000","last_active":"2024-05-15T13:10:33.113687+0000","last_peered":"2024-05-15T13:10:33.113687+0000","last_clean":"2024-05-15T13:10:33.113687+0000","last_became_active":"2024-05-15T12:35:22.781126+0000","last_became_peered":"2024-05-15T12:35:22.781126+0000","last_unstale":"2024-05-15T13:10:33.113687+0000","last_undegraded":"2024-05-15T13:10:33.113687+0000","last_fullsized":"2024-05-15T13:10:33.113687+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'4","last_scrub_stamp":"2024-05-15T13:10:33.112006+0000","last_deep_scrub":"1251'4","last_deep_scrub_stamp":"2024-05-15T13:10:33.112006+0000","last_clean_scrub_stamp":"2024-05-15T13:10:33.112006+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:20:31.241638+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.a","version":"1251'2","reported_seq":2382,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:33.293859+0000","last_change":"2024-05-15T13:10:33.295017+0000","last_active":"2024-05-15T13:10:33.293859+0000","last_peered":"2024-05-15T13:10:33.293859+0000","last_clean":"2024-05-15T13:10:33.293859+0000","last_became_active":"2024-05-15T12:35:47.943168+0000","last_became_peered":"2024-05-15T12:35:47.943168+0000","last_unstale":"2024-05-15T13:10:33.293859+0000","last_undegraded":"2024-05-15T13:10:33.293859+0000","last_fullsized":"2024-05-15T13:10:33.293859+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:33.291240+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:33.291240+0000","last_clean_scrub_stamp":"2024-05-15T13:10:33.291240+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:59:50.440846+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.9","version":"1251'2","reported_seq":2389,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:35.369057+0000","last_change":"2024-05-15T13:10:35.370041+0000","last_active":"2024-05-15T13:10:35.369057+0000","last_peered":"2024-05-15T13:10:35.369057+0000","last_clean":"2024-05-15T13:10:35.369057+0000","last_became_active":"2024-05-15T12:35:49.106669+0000","last_became_peered":"2024-05-15T12:35:49.106669+0000","last_unstale":"2024-05-15T13:10:35.369057+0000","last_undegraded":"2024-05-15T13:10:35.369057+0000","last_fullsized":"2024-05-15T13:10:35.369057+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:35.367548+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:35.367548+0000","last_clean_scrub_stamp":"2024-05-15T13:10:35.367548+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:21:42.631073+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.8","version":"0'0","reported_seq":2392,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:34.559253+0000","last_change":"2024-05-15T13:10:34.560091+0000","last_active":"2024-05-15T13:10:34.559253+0000","last_peered":"2024-05-15T13:10:34.559253+0000","last_clean":"2024-05-15T13:10:34.559253+0000","last_became_active":"2024-05-15T12:35:27.816334+0000","last_became_peered":"2024-05-15T12:35:27.816334+0000","last_unstale":"2024-05-15T13:10:34.559253+0000","last_undegraded":"2024-05-15T13:10:34.559253+0000","last_fullsized":"2024-05-15T13:10:34.559253+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:34.557589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:34.557589+0000","last_clean_scrub_stamp":"2024-05-15T13:10:34.557589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:07:57.635634+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.7","version":"1251'2","reported_seq":2387,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:34.373011+0000","last_change":"2024-05-15T13:10:34.373899+0000","last_active":"2024-05-15T13:10:34.373011+0000","last_peered":"2024-05-15T13:10:34.373011+0000","last_clean":"2024-05-15T13:10:34.373011+0000","last_became_active":"2024-05-15T12:35:47.766045+0000","last_became_peered":"2024-05-15T12:35:47.766045+0000","last_unstale":"2024-05-15T13:10:34.373011+0000","last_undegraded":"2024-05-15T13:10:34.373011+0000","last_fullsized":"2024-05-15T13:10:34.373011+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:34.371412+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:34.371412+0000","last_clean_scrub_stamp":"2024-05-15T13:10:34.371412+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:45:40.410637+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.6","version":"0'0","reported_seq":2390,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:22.244389+0000","last_change":"2024-05-15T13:10:22.245242+0000","last_active":"2024-05-15T13:10:22.244389+0000","last_peered":"2024-05-15T13:10:22.244389+0000","last_clean":"2024-05-15T13:10:22.244389+0000","last_became_active":"2024-05-15T12:35:59.010795+0000","last_became_peered":"2024-05-15T12:35:59.010795+0000","last_unstale":"2024-05-15T13:10:22.244389+0000","last_undegraded":"2024-05-15T13:10:22.244389+0000","last_fullsized":"2024-05-15T13:10:22.244389+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:22.242692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:22.242692+0000","last_clean_scrub_stamp":"2024-05-15T13:10:22.242692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:07:11.477600+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.5","version":"1251'2","reported_seq":2373,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:32.320720+0000","last_change":"2024-05-15T13:10:32.321731+0000","last_active":"2024-05-15T13:10:32.320720+0000","last_peered":"2024-05-15T13:10:32.320720+0000","last_clean":"2024-05-15T13:10:32.320720+0000","last_became_active":"2024-05-15T12:36:31.177992+0000","last_became_peered":"2024-05-15T12:36:31.177992+0000","last_unstale":"2024-05-15T13:10:32.320720+0000","last_undegraded":"2024-05-15T13:10:32.320720+0000","last_fullsized":"2024-05-15T13:10:32.320720+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:32.319041+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:32.319041+0000","last_clean_scrub_stamp":"2024-05-15T13:10:32.319041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:08:20.581923+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.4","version":"0'0","reported_seq":2388,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:23.245685+0000","last_change":"2024-05-15T13:10:23.246607+0000","last_active":"2024-05-15T13:10:23.245685+0000","last_peered":"2024-05-15T13:10:23.245685+0000","last_clean":"2024-05-15T13:10:23.245685+0000","last_became_active":"2024-05-15T12:35:59.238121+0000","last_became_peered":"2024-05-15T12:35:59.238121+0000","last_unstale":"2024-05-15T13:10:23.245685+0000","last_undegraded":"2024-05-15T13:10:23.245685+0000","last_fullsized":"2024-05-15T13:10:23.245685+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:23.243967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:23.243967+0000","last_clean_scrub_stamp":"2024-05-15T13:10:23.243967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:02:22.323698+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.3","version":"1251'8","reported_seq":2394,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:33.426260+0000","last_change":"2024-05-15T13:10:33.427237+0000","last_active":"2024-05-15T13:10:33.426260+0000","last_peered":"2024-05-15T13:10:33.426260+0000","last_clean":"2024-05-15T13:10:33.426260+0000","last_became_active":"2024-05-15T12:35:47.639452+0000","last_became_peered":"2024-05-15T12:35:47.639452+0000","last_unstale":"2024-05-15T13:10:33.426260+0000","last_undegraded":"2024-05-15T13:10:33.426260+0000","last_fullsized":"2024-05-15T13:10:33.426260+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'8","last_scrub_stamp":"2024-05-15T13:10:33.424691+0000","last_deep_scrub":"1251'8","last_deep_scrub_stamp":"2024-05-15T13:10:33.424691+0000","last_clean_scrub_stamp":"2024-05-15T13:10:33.424691+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:51:38.905464+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":704,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.2","version":"1251'2","reported_seq":2403,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:21.345186+0000","last_change":"2024-05-15T13:10:21.346519+0000","last_active":"2024-05-15T13:10:21.345186+0000","last_peered":"2024-05-15T13:10:21.345186+0000","last_clean":"2024-05-15T13:10:21.345186+0000","last_became_active":"2024-05-15T12:35:32.699857+0000","last_became_peered":"2024-05-15T12:35:32.699857+0000","last_unstale":"2024-05-15T13:10:21.345186+0000","last_undegraded":"2024-05-15T13:10:21.345186+0000","last_fullsized":"2024-05-15T13:10:21.345186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:21.343475+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:21.343475+0000","last_clean_scrub_stamp":"2024-05-15T13:10:21.343475+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:44:28.279463+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1","version":"1251'6","reported_seq":2379,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:30.302050+0000","last_change":"2024-05-15T13:10:30.303097+0000","last_active":"2024-05-15T13:10:30.302050+0000","last_peered":"2024-05-15T13:10:30.302050+0000","last_clean":"2024-05-15T13:10:30.302050+0000","last_became_active":"2024-05-15T12:36:13.614839+0000","last_became_peered":"2024-05-15T12:36:13.614839+0000","last_unstale":"2024-05-15T13:10:30.302050+0000","last_undegraded":"2024-05-15T13:10:30.302050+0000","last_fullsized":"2024-05-15T13:10:30.302050+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'6","last_scrub_stamp":"2024-05-15T13:10:30.300366+0000","last_deep_scrub":"1251'6","last_deep_scrub_stamp":"2024-05-15T13:10:30.300366+0000","last_clean_scrub_stamp":"2024-05-15T13:10:30.300366+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:14:26.643648+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.0","version":"1251'2","reported_seq":2370,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:36.127316+0000","last_change":"2024-05-15T13:10:36.128288+0000","last_active":"2024-05-15T13:10:36.127316+0000","last_peered":"2024-05-15T13:10:36.127316+0000","last_clean":"2024-05-15T13:10:36.127316+0000","last_became_active":"2024-05-15T12:35:47.271050+0000","last_became_peered":"2024-05-15T12:35:47.271050+0000","last_unstale":"2024-05-15T13:10:36.127316+0000","last_undegraded":"2024-05-15T13:10:36.127316+0000","last_fullsized":"2024-05-15T13:10:36.127316+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'2","last_scrub_stamp":"2024-05-15T13:10:36.125661+0000","last_deep_scrub":"1251'2","last_deep_scrub_stamp":"2024-05-15T13:10:36.125661+0000","last_clean_scrub_stamp":"2024-05-15T13:10:36.125661+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:59:32.773742+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":2414,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:26.228015+0000","last_change":"2024-05-15T13:10:26.229165+0000","last_active":"2024-05-15T13:10:26.228015+0000","last_peered":"2024-05-15T13:10:26.228015+0000","last_clean":"2024-05-15T13:10:26.228015+0000","last_became_active":"2024-05-15T12:34:57.154951+0000","last_became_peered":"2024-05-15T12:34:57.154951+0000","last_unstale":"2024-05-15T13:10:26.228015+0000","last_undegraded":"2024-05-15T13:10:26.228015+0000","last_fullsized":"2024-05-15T13:10:26.228015+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:26.222812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:26.222812+0000","last_clean_scrub_stamp":"2024-05-15T13:10:26.222812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:41:18.910613+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":2416,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:25.229148+0000","last_change":"2024-05-15T13:10:25.230026+0000","last_active":"2024-05-15T13:10:25.229148+0000","last_peered":"2024-05-15T13:10:25.229148+0000","last_clean":"2024-05-15T13:10:25.229148+0000","last_became_active":"2024-05-15T12:34:54.498165+0000","last_became_peered":"2024-05-15T12:34:54.498165+0000","last_unstale":"2024-05-15T13:10:25.229148+0000","last_undegraded":"2024-05-15T13:10:25.229148+0000","last_fullsized":"2024-05-15T13:10:25.229148+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:25.227391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:25.227391+0000","last_clean_scrub_stamp":"2024-05-15T13:10:25.227391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:14:55.309657+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":2423,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:32.593913+0000","last_change":"2024-05-15T13:10:32.593913+0000","last_active":"2024-05-15T13:10:32.593913+0000","last_peered":"2024-05-15T13:10:32.593913+0000","last_clean":"2024-05-15T13:10:32.593913+0000","last_became_active":"2024-05-15T12:34:52.677394+0000","last_became_peered":"2024-05-15T12:34:52.677394+0000","last_unstale":"2024-05-15T13:10:32.593913+0000","last_undegraded":"2024-05-15T13:10:32.593913+0000","last_fullsized":"2024-05-15T13:10:32.593913+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:32.591432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:32.591432+0000","last_clean_scrub_stamp":"2024-05-15T13:10:32.591432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:47:35.505251+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":2428,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:29.360211+0000","last_change":"2024-05-15T13:10:29.361001+0000","last_active":"2024-05-15T13:10:29.360211+0000","last_peered":"2024-05-15T13:10:29.360211+0000","last_clean":"2024-05-15T13:10:29.360211+0000","last_became_active":"2024-05-15T12:34:50.114215+0000","last_became_peered":"2024-05-15T12:34:50.114215+0000","last_unstale":"2024-05-15T13:10:29.360211+0000","last_undegraded":"2024-05-15T13:10:29.360211+0000","last_fullsized":"2024-05-15T13:10:29.360211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:29.358678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:29.358678+0000","last_clean_scrub_stamp":"2024-05-15T13:10:29.358678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:17:35.758984+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":2427,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:26.041113+0000","last_change":"2024-05-15T13:10:26.041113+0000","last_active":"2024-05-15T13:10:26.041113+0000","last_peered":"2024-05-15T13:10:26.041113+0000","last_clean":"2024-05-15T13:10:26.041113+0000","last_became_active":"2024-05-15T12:34:54.762972+0000","last_became_peered":"2024-05-15T12:34:54.762972+0000","last_unstale":"2024-05-15T13:10:26.041113+0000","last_undegraded":"2024-05-15T13:10:26.041113+0000","last_fullsized":"2024-05-15T13:10:26.041113+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:26.038846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:26.038846+0000","last_clean_scrub_stamp":"2024-05-15T13:10:26.038846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:06:45.110912+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":2430,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:26.205182+0000","last_change":"2024-05-15T13:10:26.205952+0000","last_active":"2024-05-15T13:10:26.205182+0000","last_peered":"2024-05-15T13:10:26.205182+0000","last_clean":"2024-05-15T13:10:26.205182+0000","last_became_active":"2024-05-15T12:34:34.670411+0000","last_became_peered":"2024-05-15T12:34:34.670411+0000","last_unstale":"2024-05-15T13:10:26.205182+0000","last_undegraded":"2024-05-15T13:10:26.205182+0000","last_fullsized":"2024-05-15T13:10:26.205182+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:26.203699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:26.203699+0000","last_clean_scrub_stamp":"2024-05-15T13:10:26.203699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:21:53.243567+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":2429,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:30.333544+0000","last_change":"2024-05-15T13:10:30.334315+0000","last_active":"2024-05-15T13:10:30.333544+0000","last_peered":"2024-05-15T13:10:30.333544+0000","last_clean":"2024-05-15T13:10:30.333544+0000","last_became_active":"2024-05-15T12:34:50.159068+0000","last_became_peered":"2024-05-15T12:34:50.159068+0000","last_unstale":"2024-05-15T13:10:30.333544+0000","last_undegraded":"2024-05-15T13:10:30.333544+0000","last_fullsized":"2024-05-15T13:10:30.333544+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:30.331894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:30.331894+0000","last_clean_scrub_stamp":"2024-05-15T13:10:30.331894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:17:23.969226+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":2435,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:26.324655+0000","last_change":"2024-05-15T13:10:26.325404+0000","last_active":"2024-05-15T13:10:26.324655+0000","last_peered":"2024-05-15T13:10:26.324655+0000","last_clean":"2024-05-15T13:10:26.324655+0000","last_became_active":"2024-05-15T12:34:39.907050+0000","last_became_peered":"2024-05-15T12:34:39.907050+0000","last_unstale":"2024-05-15T13:10:26.324655+0000","last_undegraded":"2024-05-15T13:10:26.324655+0000","last_fullsized":"2024-05-15T13:10:26.324655+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:26.323130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:26.323130+0000","last_clean_scrub_stamp":"2024-05-15T13:10:26.323130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:41:15.923524+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":2409,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:32.032463+0000","last_change":"2024-05-15T13:10:32.033253+0000","last_active":"2024-05-15T13:10:32.032463+0000","last_peered":"2024-05-15T13:10:32.032463+0000","last_clean":"2024-05-15T13:10:32.032463+0000","last_became_active":"2024-05-15T12:34:44.443445+0000","last_became_peered":"2024-05-15T12:34:44.443445+0000","last_unstale":"2024-05-15T13:10:32.032463+0000","last_undegraded":"2024-05-15T13:10:32.032463+0000","last_fullsized":"2024-05-15T13:10:32.032463+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:32.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:32.030814+0000","last_clean_scrub_stamp":"2024-05-15T13:10:32.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:05:24.252773+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":2411,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:28.975910+0000","last_change":"2024-05-15T13:10:28.976723+0000","last_active":"2024-05-15T13:10:28.975910+0000","last_peered":"2024-05-15T13:10:28.975910+0000","last_clean":"2024-05-15T13:10:28.975910+0000","last_became_active":"2024-05-15T12:34:38.194477+0000","last_became_peered":"2024-05-15T12:34:38.194477+0000","last_unstale":"2024-05-15T13:10:28.975910+0000","last_undegraded":"2024-05-15T13:10:28.975910+0000","last_fullsized":"2024-05-15T13:10:28.975910+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:28.974366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:28.974366+0000","last_clean_scrub_stamp":"2024-05-15T13:10:28.974366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:39:33.073332+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":2416,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:24.224104+0000","last_change":"2024-05-15T13:10:24.224104+0000","last_active":"2024-05-15T13:10:24.224104+0000","last_peered":"2024-05-15T13:10:24.224104+0000","last_clean":"2024-05-15T13:10:24.224104+0000","last_became_active":"2024-05-15T12:34:42.317054+0000","last_became_peered":"2024-05-15T12:34:42.317054+0000","last_unstale":"2024-05-15T13:10:24.224104+0000","last_undegraded":"2024-05-15T13:10:24.224104+0000","last_fullsized":"2024-05-15T13:10:24.224104+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:24.221570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:24.221570+0000","last_clean_scrub_stamp":"2024-05-15T13:10:24.221570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:42:26.838557+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":2425,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:23.034432+0000","last_change":"2024-05-15T13:10:23.034432+0000","last_active":"2024-05-15T13:10:23.034432+0000","last_peered":"2024-05-15T13:10:23.034432+0000","last_clean":"2024-05-15T13:10:23.034432+0000","last_became_active":"2024-05-15T12:34:48.742068+0000","last_became_peered":"2024-05-15T12:34:48.742068+0000","last_unstale":"2024-05-15T13:10:23.034432+0000","last_undegraded":"2024-05-15T13:10:23.034432+0000","last_fullsized":"2024-05-15T13:10:23.034432+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:23.030600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:23.030600+0000","last_clean_scrub_stamp":"2024-05-15T13:10:23.030600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:38:37.379516+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"1251'78","reported_seq":2478,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:21.334748+0000","last_change":"2024-05-15T13:10:21.336063+0000","last_active":"2024-05-15T13:10:21.334748+0000","last_peered":"2024-05-15T13:10:21.334748+0000","last_clean":"2024-05-15T13:10:21.334748+0000","last_became_active":"2024-05-15T12:32:42.889644+0000","last_became_peered":"2024-05-15T12:32:42.889644+0000","last_unstale":"2024-05-15T13:10:21.334748+0000","last_undegraded":"2024-05-15T13:10:21.334748+0000","last_fullsized":"2024-05-15T13:10:21.334748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'78","last_scrub_stamp":"2024-05-15T13:10:21.333003+0000","last_deep_scrub":"1251'78","last_deep_scrub_stamp":"2024-05-15T13:10:21.333003+0000","last_clean_scrub_stamp":"2024-05-15T13:10:21.333003+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:22:57.058726+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2438,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:26.597340+0000","last_change":"2024-05-15T13:10:26.597340+0000","last_active":"2024-05-15T13:10:26.597340+0000","last_peered":"2024-05-15T13:10:26.597340+0000","last_clean":"2024-05-15T13:10:26.597340+0000","last_became_active":"2024-05-15T12:32:47.716674+0000","last_became_peered":"2024-05-15T12:32:47.716674+0000","last_unstale":"2024-05-15T13:10:26.597340+0000","last_undegraded":"2024-05-15T13:10:26.597340+0000","last_fullsized":"2024-05-15T13:10:26.597340+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:26.594894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:26.594894+0000","last_clean_scrub_stamp":"2024-05-15T13:10:26.594894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:27:00.922496+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":2434,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:27.363771+0000","last_change":"2024-05-15T13:10:27.364522+0000","last_active":"2024-05-15T13:10:27.363771+0000","last_peered":"2024-05-15T13:10:27.363771+0000","last_clean":"2024-05-15T13:10:27.363771+0000","last_became_active":"2024-05-15T12:34:42.942214+0000","last_became_peered":"2024-05-15T12:34:42.942214+0000","last_unstale":"2024-05-15T13:10:27.363771+0000","last_undegraded":"2024-05-15T13:10:27.363771+0000","last_fullsized":"2024-05-15T13:10:27.363771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:27.362216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:27.362216+0000","last_clean_scrub_stamp":"2024-05-15T13:10:27.362216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:55:57.634138+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2438,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:23.508417+0000","last_change":"2024-05-15T13:10:23.509292+0000","last_active":"2024-05-15T13:10:23.508417+0000","last_peered":"2024-05-15T13:10:23.508417+0000","last_clean":"2024-05-15T13:10:23.508417+0000","last_became_active":"2024-05-15T12:32:47.430194+0000","last_became_peered":"2024-05-15T12:32:47.430194+0000","last_unstale":"2024-05-15T13:10:23.508417+0000","last_undegraded":"2024-05-15T13:10:23.508417+0000","last_fullsized":"2024-05-15T13:10:23.508417+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:23.506650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:23.506650+0000","last_clean_scrub_stamp":"2024-05-15T13:10:23.506650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:08:06.263109+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":2425,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:31.606122+0000","last_change":"2024-05-15T13:10:31.606988+0000","last_active":"2024-05-15T13:10:31.606122+0000","last_peered":"2024-05-15T13:10:31.606122+0000","last_clean":"2024-05-15T13:10:31.606122+0000","last_became_active":"2024-05-15T12:34:52.357328+0000","last_became_peered":"2024-05-15T12:34:52.357328+0000","last_unstale":"2024-05-15T13:10:31.606122+0000","last_undegraded":"2024-05-15T13:10:31.606122+0000","last_fullsized":"2024-05-15T13:10:31.606122+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:31.604503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:31.604503+0000","last_clean_scrub_stamp":"2024-05-15T13:10:31.604503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:05:14.518611+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2425,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:26.917103+0000","last_change":"2024-05-15T13:10:26.917928+0000","last_active":"2024-05-15T13:10:26.917103+0000","last_peered":"2024-05-15T13:10:26.917103+0000","last_clean":"2024-05-15T13:10:26.917103+0000","last_became_active":"2024-05-15T12:32:47.432679+0000","last_became_peered":"2024-05-15T12:32:47.432679+0000","last_unstale":"2024-05-15T13:10:26.917103+0000","last_undegraded":"2024-05-15T13:10:26.917103+0000","last_fullsized":"2024-05-15T13:10:26.917103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:26.915501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:26.915501+0000","last_clean_scrub_stamp":"2024-05-15T13:10:26.915501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:08:54.655971+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":2431,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:28.221564+0000","last_change":"2024-05-15T13:10:28.222344+0000","last_active":"2024-05-15T13:10:28.221564+0000","last_peered":"2024-05-15T13:10:28.221564+0000","last_clean":"2024-05-15T13:10:28.221564+0000","last_became_active":"2024-05-15T12:34:36.924450+0000","last_became_peered":"2024-05-15T12:34:36.924450+0000","last_unstale":"2024-05-15T13:10:28.221564+0000","last_undegraded":"2024-05-15T13:10:28.221564+0000","last_fullsized":"2024-05-15T13:10:28.221564+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:28.220076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:28.220076+0000","last_clean_scrub_stamp":"2024-05-15T13:10:28.220076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:02:58.252383+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2441,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:25.569617+0000","last_change":"2024-05-15T13:10:25.569617+0000","last_active":"2024-05-15T13:10:25.569617+0000","last_peered":"2024-05-15T13:10:25.569617+0000","last_clean":"2024-05-15T13:10:25.569617+0000","last_became_active":"2024-05-15T12:32:47.670226+0000","last_became_peered":"2024-05-15T12:32:47.670226+0000","last_unstale":"2024-05-15T13:10:25.569617+0000","last_undegraded":"2024-05-15T13:10:25.569617+0000","last_fullsized":"2024-05-15T13:10:25.569617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:25.567130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:25.567130+0000","last_clean_scrub_stamp":"2024-05-15T13:10:25.567130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:35:01.836446+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":2424,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:30.581611+0000","last_change":"2024-05-15T13:10:30.582457+0000","last_active":"2024-05-15T13:10:30.581611+0000","last_peered":"2024-05-15T13:10:30.581611+0000","last_clean":"2024-05-15T13:10:30.581611+0000","last_became_active":"2024-05-15T12:34:51.746853+0000","last_became_peered":"2024-05-15T12:34:51.746853+0000","last_unstale":"2024-05-15T13:10:30.581611+0000","last_undegraded":"2024-05-15T13:10:30.581611+0000","last_fullsized":"2024-05-15T13:10:30.581611+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:30.579935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:30.579935+0000","last_clean_scrub_stamp":"2024-05-15T13:10:30.579935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:55:34.647689+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2448,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:25.331423+0000","last_change":"2024-05-15T13:10:25.331423+0000","last_active":"2024-05-15T13:10:25.331423+0000","last_peered":"2024-05-15T13:10:25.331423+0000","last_clean":"2024-05-15T13:10:25.331423+0000","last_became_active":"2024-05-15T12:32:47.680498+0000","last_became_peered":"2024-05-15T12:32:47.680498+0000","last_unstale":"2024-05-15T13:10:25.331423+0000","last_undegraded":"2024-05-15T13:10:25.331423+0000","last_fullsized":"2024-05-15T13:10:25.331423+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:25.329043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:25.329043+0000","last_clean_scrub_stamp":"2024-05-15T13:10:25.329043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:41:28.317152+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"1251'8","reported_seq":2441,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:24.073755+0000","last_change":"2024-05-15T13:10:24.075097+0000","last_active":"2024-05-15T13:10:24.073755+0000","last_peered":"2024-05-15T13:10:24.073755+0000","last_clean":"2024-05-15T13:10:24.073755+0000","last_became_active":"2024-05-15T12:34:50.371773+0000","last_became_peered":"2024-05-15T12:34:50.371773+0000","last_unstale":"2024-05-15T13:10:24.073755+0000","last_undegraded":"2024-05-15T13:10:24.073755+0000","last_fullsized":"2024-05-15T13:10:24.073755+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"1251'8","last_scrub_stamp":"2024-05-15T13:10:24.072030+0000","last_deep_scrub":"1251'8","last_deep_scrub_stamp":"2024-05-15T13:10:24.072030+0000","last_clean_scrub_stamp":"2024-05-15T13:10:24.072030+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:38:41.763044+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2437,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:24.558776+0000","last_change":"2024-05-15T13:10:24.558776+0000","last_active":"2024-05-15T13:10:24.558776+0000","last_peered":"2024-05-15T13:10:24.558776+0000","last_clean":"2024-05-15T13:10:24.558776+0000","last_became_active":"2024-05-15T12:32:47.526275+0000","last_became_peered":"2024-05-15T12:32:47.526275+0000","last_unstale":"2024-05-15T13:10:24.558776+0000","last_undegraded":"2024-05-15T13:10:24.558776+0000","last_fullsized":"2024-05-15T13:10:24.558776+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:24.553935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:24.553935+0000","last_clean_scrub_stamp":"2024-05-15T13:10:24.553935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:47:03.395862+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":2419,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:23.238174+0000","last_change":"2024-05-15T13:10:23.239057+0000","last_active":"2024-05-15T13:10:23.238174+0000","last_peered":"2024-05-15T13:10:23.238174+0000","last_clean":"2024-05-15T13:10:23.238174+0000","last_became_active":"2024-05-15T12:34:38.888318+0000","last_became_peered":"2024-05-15T12:34:38.888318+0000","last_unstale":"2024-05-15T13:10:23.238174+0000","last_undegraded":"2024-05-15T13:10:23.238174+0000","last_fullsized":"2024-05-15T13:10:23.238174+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:23.235879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:23.235879+0000","last_clean_scrub_stamp":"2024-05-15T13:10:23.235879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T01:07:36.570458+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2441,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:24.187198+0000","last_change":"2024-05-15T13:10:24.188045+0000","last_active":"2024-05-15T13:10:24.187198+0000","last_peered":"2024-05-15T13:10:24.187198+0000","last_clean":"2024-05-15T13:10:24.187198+0000","last_became_active":"2024-05-15T12:32:48.283635+0000","last_became_peered":"2024-05-15T12:32:48.283635+0000","last_unstale":"2024-05-15T13:10:24.187198+0000","last_undegraded":"2024-05-15T13:10:24.187198+0000","last_fullsized":"2024-05-15T13:10:24.187198+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:24.185536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:24.185536+0000","last_clean_scrub_stamp":"2024-05-15T13:10:24.185536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:31:29.217330+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":2435,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:28.381103+0000","last_change":"2024-05-15T13:10:28.381930+0000","last_active":"2024-05-15T13:10:28.381103+0000","last_peered":"2024-05-15T13:10:28.381103+0000","last_clean":"2024-05-15T13:10:28.381103+0000","last_became_active":"2024-05-15T12:34:48.664879+0000","last_became_peered":"2024-05-15T12:34:48.664879+0000","last_unstale":"2024-05-15T13:10:28.381103+0000","last_undegraded":"2024-05-15T13:10:28.381103+0000","last_fullsized":"2024-05-15T13:10:28.381103+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:28.379098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:28.379098+0000","last_clean_scrub_stamp":"2024-05-15T13:10:28.379098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:39:41.025914+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2435,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:27.565146+0000","last_change":"2024-05-15T13:10:27.565984+0000","last_active":"2024-05-15T13:10:27.565146+0000","last_peered":"2024-05-15T13:10:27.565146+0000","last_clean":"2024-05-15T13:10:27.565146+0000","last_became_active":"2024-05-15T12:32:48.252725+0000","last_became_peered":"2024-05-15T12:32:48.252725+0000","last_unstale":"2024-05-15T13:10:27.565146+0000","last_undegraded":"2024-05-15T13:10:27.565146+0000","last_fullsized":"2024-05-15T13:10:27.565146+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:27.563531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:27.563531+0000","last_clean_scrub_stamp":"2024-05-15T13:10:27.563531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:40:17.834143+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":2431,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:25.207390+0000","last_change":"2024-05-15T13:10:25.208158+0000","last_active":"2024-05-15T13:10:25.207390+0000","last_peered":"2024-05-15T13:10:25.207390+0000","last_clean":"2024-05-15T13:10:25.207390+0000","last_became_active":"2024-05-15T12:34:34.660678+0000","last_became_peered":"2024-05-15T12:34:34.660678+0000","last_unstale":"2024-05-15T13:10:25.207390+0000","last_undegraded":"2024-05-15T13:10:25.207390+0000","last_fullsized":"2024-05-15T13:10:25.207390+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:25.205828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:25.205828+0000","last_clean_scrub_stamp":"2024-05-15T13:10:25.205828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:40:38.471407+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":2411,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:30.009779+0000","last_change":"2024-05-15T13:10:30.010548+0000","last_active":"2024-05-15T13:10:30.009779+0000","last_peered":"2024-05-15T13:10:30.009779+0000","last_clean":"2024-05-15T13:10:30.009779+0000","last_became_active":"2024-05-15T12:34:44.277750+0000","last_became_peered":"2024-05-15T12:34:44.277750+0000","last_unstale":"2024-05-15T13:10:30.009779+0000","last_undegraded":"2024-05-15T13:10:30.009779+0000","last_fullsized":"2024-05-15T13:10:30.009779+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:30.008154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:30.008154+0000","last_clean_scrub_stamp":"2024-05-15T13:10:30.008154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:11:21.319890+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":2425,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:22.008821+0000","last_change":"2024-05-15T13:10:22.009696+0000","last_active":"2024-05-15T13:10:22.008821+0000","last_peered":"2024-05-15T13:10:22.008821+0000","last_clean":"2024-05-15T13:10:22.008821+0000","last_became_active":"2024-05-15T12:34:39.369183+0000","last_became_peered":"2024-05-15T12:34:39.369183+0000","last_unstale":"2024-05-15T13:10:22.008821+0000","last_undegraded":"2024-05-15T13:10:22.008821+0000","last_fullsized":"2024-05-15T13:10:22.008821+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:22.006994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:22.006994+0000","last_clean_scrub_stamp":"2024-05-15T13:10:22.006994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:29:40.757842+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":2417,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:22.206465+0000","last_change":"2024-05-15T13:10:22.207331+0000","last_active":"2024-05-15T13:10:22.206465+0000","last_peered":"2024-05-15T13:10:22.206465+0000","last_clean":"2024-05-15T13:10:22.206465+0000","last_became_active":"2024-05-15T12:34:38.812314+0000","last_became_peered":"2024-05-15T12:34:38.812314+0000","last_unstale":"2024-05-15T13:10:22.206465+0000","last_undegraded":"2024-05-15T13:10:22.206465+0000","last_fullsized":"2024-05-15T13:10:22.206465+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:22.204826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:22.204826+0000","last_clean_scrub_stamp":"2024-05-15T13:10:22.204826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:26:09.187859+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":2410,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:27.966914+0000","last_change":"2024-05-15T13:10:27.967682+0000","last_active":"2024-05-15T13:10:27.966914+0000","last_peered":"2024-05-15T13:10:27.966914+0000","last_clean":"2024-05-15T13:10:27.966914+0000","last_became_active":"2024-05-15T12:34:36.777340+0000","last_became_peered":"2024-05-15T12:34:36.777340+0000","last_unstale":"2024-05-15T13:10:27.966914+0000","last_undegraded":"2024-05-15T13:10:27.966914+0000","last_fullsized":"2024-05-15T13:10:27.966914+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:27.965358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:27.965358+0000","last_clean_scrub_stamp":"2024-05-15T13:10:27.965358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:39:14.825494+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":2423,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:25.025027+0000","last_change":"2024-05-15T13:10:25.025873+0000","last_active":"2024-05-15T13:10:25.025027+0000","last_peered":"2024-05-15T13:10:25.025027+0000","last_clean":"2024-05-15T13:10:25.025027+0000","last_became_active":"2024-05-15T12:34:51.040048+0000","last_became_peered":"2024-05-15T12:34:51.040048+0000","last_unstale":"2024-05-15T13:10:25.025027+0000","last_undegraded":"2024-05-15T13:10:25.025027+0000","last_fullsized":"2024-05-15T13:10:25.025027+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:25.023373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:25.023373+0000","last_clean_scrub_stamp":"2024-05-15T13:10:25.023373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:13:49.717646+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":2433,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:31.364774+0000","last_change":"2024-05-15T13:10:31.365566+0000","last_active":"2024-05-15T13:10:31.364774+0000","last_peered":"2024-05-15T13:10:31.364774+0000","last_clean":"2024-05-15T13:10:31.364774+0000","last_became_active":"2024-05-15T12:34:51.649910+0000","last_became_peered":"2024-05-15T12:34:51.649910+0000","last_unstale":"2024-05-15T13:10:31.364774+0000","last_undegraded":"2024-05-15T13:10:31.364774+0000","last_fullsized":"2024-05-15T13:10:31.364774+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:31.363191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:31.363191+0000","last_clean_scrub_stamp":"2024-05-15T13:10:31.363191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:15:04.158501+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":2413,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:31.023344+0000","last_change":"2024-05-15T13:10:31.024108+0000","last_active":"2024-05-15T13:10:31.023344+0000","last_peered":"2024-05-15T13:10:31.023344+0000","last_clean":"2024-05-15T13:10:31.023344+0000","last_became_active":"2024-05-15T12:34:44.399991+0000","last_became_peered":"2024-05-15T12:34:44.399991+0000","last_unstale":"2024-05-15T13:10:31.023344+0000","last_undegraded":"2024-05-15T13:10:31.023344+0000","last_fullsized":"2024-05-15T13:10:31.023344+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:31.021797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:31.021797+0000","last_clean_scrub_stamp":"2024-05-15T13:10:31.021797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:49:11.725401+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":2437,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:20.323634+0000","last_change":"2024-05-15T13:10:20.324517+0000","last_active":"2024-05-15T13:10:20.323634+0000","last_peered":"2024-05-15T13:10:20.323634+0000","last_clean":"2024-05-15T13:10:20.323634+0000","last_became_active":"2024-05-15T12:34:38.549385+0000","last_became_peered":"2024-05-15T12:34:38.549385+0000","last_unstale":"2024-05-15T13:10:20.323634+0000","last_undegraded":"2024-05-15T13:10:20.323634+0000","last_fullsized":"2024-05-15T13:10:20.323634+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:20.318383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:20.318383+0000","last_clean_scrub_stamp":"2024-05-15T13:10:20.318383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:42:59.241875+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":2428,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:27.246326+0000","last_change":"2024-05-15T13:10:27.247141+0000","last_active":"2024-05-15T13:10:27.246326+0000","last_peered":"2024-05-15T13:10:27.246326+0000","last_clean":"2024-05-15T13:10:27.246326+0000","last_became_active":"2024-05-15T12:34:36.875596+0000","last_became_peered":"2024-05-15T12:34:36.875596+0000","last_unstale":"2024-05-15T13:10:27.246326+0000","last_undegraded":"2024-05-15T13:10:27.246326+0000","last_fullsized":"2024-05-15T13:10:27.246326+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:27.244825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:27.244825+0000","last_clean_scrub_stamp":"2024-05-15T13:10:27.244825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:02:36.937199+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":2424,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:29.596841+0000","last_change":"2024-05-15T13:10:29.597679+0000","last_active":"2024-05-15T13:10:29.596841+0000","last_peered":"2024-05-15T13:10:29.596841+0000","last_clean":"2024-05-15T13:10:29.596841+0000","last_became_active":"2024-05-15T12:34:51.653558+0000","last_became_peered":"2024-05-15T12:34:51.653558+0000","last_unstale":"2024-05-15T13:10:29.596841+0000","last_undegraded":"2024-05-15T13:10:29.596841+0000","last_fullsized":"2024-05-15T13:10:29.596841+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:29.595211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:29.595211+0000","last_clean_scrub_stamp":"2024-05-15T13:10:29.595211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:47:59.053842+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":2424,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:28.607693+0000","last_change":"2024-05-15T13:10:28.608533+0000","last_active":"2024-05-15T13:10:28.607693+0000","last_peered":"2024-05-15T13:10:28.607693+0000","last_clean":"2024-05-15T13:10:28.607693+0000","last_became_active":"2024-05-15T12:34:36.629372+0000","last_became_peered":"2024-05-15T12:34:36.629372+0000","last_unstale":"2024-05-15T13:10:28.607693+0000","last_undegraded":"2024-05-15T13:10:28.607693+0000","last_fullsized":"2024-05-15T13:10:28.607693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:28.606040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:28.606040+0000","last_clean_scrub_stamp":"2024-05-15T13:10:28.606040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:51:42.364760+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":2425,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-05-15T13:10:29.247880+0000","last_change":"2024-05-15T13:10:29.248648+0000","last_active":"2024-05-15T13:10:29.247880+0000","last_peered":"2024-05-15T13:10:29.247880+0000","last_clean":"2024-05-15T13:10:29.247880+0000","last_became_active":"2024-05-15T12:34:49.324761+0000","last_became_peered":"2024-05-15T12:34:49.324761+0000","last_unstale":"2024-05-15T13:10:29.247880+0000","last_undegraded":"2024-05-15T13:10:29.247880+0000","last_fullsized":"2024-05-15T13:10:29.247880+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T13:10:29.246396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T13:10:29.246396+0000","last_clean_scrub_stamp":"2024-05-15T13:10:29.246396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:58:01.334279+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":6703,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":13406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":64,"acting":64,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":336400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313674,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68852,"kb_used_data":5740,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302988,"statfs":{"total":96636764160,"available":96566259712,"internally_reserved":0,"allocated":5877760,"data_stored":3157052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":19,"seq":81604379079,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72572,"kb_used_data":5300,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299268,"statfs":{"total":96636764160,"available":96562450432,"internally_reserved":0,"allocated":5427200,"data_stored":2757884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604379086,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72984,"kb_used_data":5712,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298856,"statfs":{"total":96636764160,"available":96562028544,"internally_reserved":0,"allocated":5849088,"data_stored":3156580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444491,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76740,"kb_used_data":5308,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295100,"statfs":{"total":96636764160,"available":96558182400,"internally_reserved":0,"allocated":5435392,"data_stored":2757816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444489,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72544,"kb_used_data":5272,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299296,"statfs":{"total":96636764160,"available":96562479104,"internally_reserved":0,"allocated":5398528,"data_stored":2756889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542605,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72600,"kb_used_data":5328,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299240,"statfs":{"total":96636764160,"available":96562421760,"internally_reserved":0,"allocated":5455872,"data_stored":2759133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509896,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68420,"kb_used_data":5308,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303420,"statfs":{"total":96636764160,"available":96566702080,"internally_reserved":0,"allocated":5435392,"data_stored":2758174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575307,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76736,"kb_used_data":5304,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295104,"statfs":{"total":96636764160,"available":96558186496,"internally_reserved":0,"allocated":5431296,"data_stored":2757936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":2587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T13:10:47.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-15T13:10:47.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.610+0000 7f4f8242c640 10 monclient: get_monmap_and_config 2024-05-15T13:10:47.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.610+0000 7f4f8242c640 10 monclient: build_initial_monmap 2024-05-15T13:10:47.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.610+0000 7f4f8242c640 10 monclient: monmap: 2024-05-15T13:10:47.601 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:47.601 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:47.601 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T13:10:47.611716+0000 2024-05-15T13:10:47.601 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T13:10:47.611716+0000 2024-05-15T13:10:47.602 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:47.602 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:47.602 INFO:teuthology.orchestra.run.smithi049.stderr:0: v1:172.21.15.49:6789/0 mon.noname-a-legacy 2024-05-15T13:10:47.602 INFO:teuthology.orchestra.run.smithi049.stderr:1: v1:172.21.15.88:6789/0 mon.noname-c-legacy 2024-05-15T13:10:47.602 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-b 2024-05-15T13:10:47.602 INFO:teuthology.orchestra.run.smithi049.stderr:3: v2:172.21.15.49:3300/0 mon.noname-a 2024-05-15T13:10:47.602 INFO:teuthology.orchestra.run.smithi049.stderr:4: v2:172.21.15.88:3300/0 mon.noname-c 2024-05-15T13:10:47.602 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:47.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.610+0000 7f4f8242c640 1 Processor -- start 2024-05-15T13:10:47.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 1 -- start start 2024-05-15T13:10:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 10 monclient: init 2024-05-15T13:10:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-05-15T13:10:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c0b53e0 0x7f4f7c0a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 10 monclient(hunting): picked mon.noname-b con 0x7f4f7c0b53e0 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f7c142870 addr v1:172.21.15.49:6789/0 2024-05-15T13:10:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f7c13f660 addr v1:172.21.15.88:6789/0 2024-05-15T13:10:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c0a9f30 con 0x7f4f7c142870 2024-05-15T13:10:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c0ab230 con 0x7f4f7c13f660 2024-05-15T13:10:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c0b57c0 con 0x7f4f7c0b53e0 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.611+0000 7f4f8242c640 10 monclient(hunting): authenticate will time out at 3805.774084s 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.612+0000 7f4f7b7fe640 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f4f7c142870 0x7f4f7c142cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:48536/0 (socket says 172.21.15.49:48536) 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.612+0000 7f4f7b7fe640 1 -- 172.21.15.49:0/1444440442 learned_addr learned my addr 172.21.15.49:0/1444440442 (peer_addr_for_me v1:172.21.15.49:0/0) 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.612+0000 7f4f7bfff640 1 --2- 172.21.15.49:0/1444440442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c0b53e0 0x7f4f7c0a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.612+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f7c0b53e0 auth_method 0 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.612+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.612+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.612+0000 7f4f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.612+0000 7f4f809c9640 1 -- 172.21.15.49:0/1444440442 >> v1:172.21.15.88:6789/0 conn(0x7f4f7c13f660 legacy=0x7f4f7c13fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.612+0000 7f4f809c9640 1 -- 172.21.15.49:0/1444440442 >> v1:172.21.15.88:6789/0 conn(0x7f4f7c13f660 legacy=0x7f4f7c13fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.612+0000 7f4f809c9640 1 --1- 172.21.15.49:0/1444440442 >> v1:172.21.15.88:6789/0 conn(0x7f4f7c13f660 0x7f4f7c13fad0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-05-15T13:10:47.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.613+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:47.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.613+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:47.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.613+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.614+0000 7f4f7affd640 1 -- 172.21.15.49:0/1444440442 <== mon.0 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106036974 0 0) 0x7f4f7c0a9f30 con 0x7f4f7c142870 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.614+0000 7f4f7affd640 10 monclient(hunting): _negotiate 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.614+0000 7f4f7affd640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.614+0000 7f4f7affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.614+0000 7f4f7affd640 10 monclient(hunting): my global_id is 16254 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.614+0000 7f4f7affd640 1 -- 172.21.15.49:0/1444440442 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f64003520 con 0x7f4f7c142870 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.615+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_done global_id 17678 payload 274 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.615+0000 7f4f7bfff640 1 -- 172.21.15.49:0/1444440442 >> v1:172.21.15.88:6789/0 conn(0x7f4f7c13f660 legacy=0x7f4f7c13fad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.615+0000 7f4f7bfff640 1 -- 172.21.15.49:0/1444440442 >> v1:172.21.15.49:6789/0 conn(0x7f4f7c142870 legacy=0x7f4f7c142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.615+0000 7f4f7bfff640 10 monclient: _finish_hunting 0 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.615+0000 7f4f7bfff640 1 monclient: found mon.noname-b 2024-05-15T13:10:47.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.615+0000 7f4f7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:47.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.615+0000 7f4f7bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.49:3301/0 2024-05-15T13:10:47.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.615+0000 7f4f7bfff640 1 -- 172.21.15.49:0/1444440442 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0ab510 con 0x7f4f7c0b53e0 2024-05-15T13:10:47.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.616+0000 7f4f7bfff640 1 --2- 172.21.15.49:0/1444440442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c0b53e0 0x7f4f7c0a99f0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0097d0 tx=0x7f4f6c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af2bc5ef08f344f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:47.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 1 -- 172.21.15.49:0/1444440442 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c031770 con 0x7f4f7c0b53e0 2024-05-15T13:10:47.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:47.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 10 monclient: dump: 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 10 monclient: _finish_auth 0 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 10 monclient: _check_auth_tickets 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 1 -- 172.21.15.49:0/1444440442 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6c037070 con 0x7f4f7c0b53e0 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f8242c640 5 monclient: authenticate success, global_id 17678 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f8242c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f8242c640 10 monclient: get_monmap_and_config success 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 1 -- 172.21.15.49:0/1444440442 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c031b30 con 0x7f4f7c0b53e0 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f8242c640 4 set_mon_vals no callback set 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.620+0000 7f4f7affd640 10 monclient: dump: 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:47.614 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.621+0000 7f4f8242c640 10 monclient: shutdown 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.621+0000 7f4f8242c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.621+0000 7f4f8242c640 1 -- 172.21.15.49:0/1444440442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c0b53e0 msgr2=0x7f4f7c0a99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.621+0000 7f4f8242c640 1 --2- 172.21.15.49:0/1444440442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c0b53e0 0x7f4f7c0a99f0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0097d0 tx=0x7f4f6c02b810 comp rx=0 tx=0).stop 2024-05-15T13:10:47.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.621+0000 7f4f8242c640 1 -- 172.21.15.49:0/1444440442 shutdown_connections 2024-05-15T13:10:47.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.621+0000 7f4f8242c640 1 --2- 172.21.15.49:0/1444440442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c0b53e0 0x7f4f7c0a99f0 unknown :-1 s=CLOSED pgs=3127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:47.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.621+0000 7f4f8242c640 1 -- 172.21.15.49:0/1444440442 >> 172.21.15.49:0/1444440442 conn(0x7f4f7c066ed0 msgr2=0x7f4f7c0672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:47.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.621+0000 7f4f8242c640 1 -- 172.21.15.49:0/1444440442 shutdown_connections 2024-05-15T13:10:47.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.621+0000 7f4f8242c640 1 -- 172.21.15.49:0/1444440442 wait complete. 2024-05-15T13:10:47.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.622+0000 7f4f8242c640 10 monclient: build_initial_monmap 2024-05-15T13:10:47.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.622+0000 7f4f8242c640 10 monclient: monmap: 2024-05-15T13:10:47.619 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 0 2024-05-15T13:10:47.619 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-15T13:10:47.619 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 0.000000 2024-05-15T13:10:47.619 INFO:teuthology.orchestra.run.smithi049.stderr:created 0.000000 2024-05-15T13:10:47.619 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 0 (unknown) 2024-05-15T13:10:47.619 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:47.619 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.noname-a 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-b 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.noname-c 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.622+0000 7f4f8242c640 1 librados: starting msgr at 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.622+0000 7f4f8242c640 1 librados: starting objecter 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.622+0000 7f4f8242c640 1 Processor -- start 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.622+0000 7f4f8242c640 1 -- start start 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.622+0000 7f4f8242c640 1 librados: setting wanted keys 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.622+0000 7f4f8242c640 1 librados: calling monclient init 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.622+0000 7f4f8242c640 10 monclient: init 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 10 monclient: _reopen_session rank -1 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f7c0b53e0 0x7f4f7c0abf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:47.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 10 monclient(hunting): picked mon.noname-b con 0x7f4f7c0b53e0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f7c13f660 0x7f4f7c0ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 10 monclient(hunting): picked mon.noname-a con 0x7f4f7c13f660 addr [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c142870 0x7f4f7c134160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 10 monclient(hunting): picked mon.noname-c con 0x7f4f7c142870 addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c1401e0 con 0x7f4f7c13f660 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c0aa260 con 0x7f4f7c0b53e0 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 10 monclient(hunting): start opening mon connection 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c140360 con 0x7f4f7c142870 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 10 monclient(hunting): _renew_subs 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f8242c640 10 monclient(hunting): authenticate will time out at 3805.785889s 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.623+0000 7f4f7b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f7c13f660 0x7f4f7c0ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:47.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f809c9640 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c142870 0x7f4f7c134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f7c0b53e0 0x7f4f7c0abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f7c13f660 0x7f4f7c0ad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56058/0 (socket says 172.21.15.49:56058) 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7b7fe640 1 -- 172.21.15.49:0/1602188201 learned_addr learned my addr 172.21.15.49:0/1602188201 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7b7fe640 10 monclient(hunting): get_auth_request con 0x7f4f7c13f660 auth_method 0 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f809c9640 10 monclient(hunting): get_auth_request con 0x7f4f7c142870 auth_method 0 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f809c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f809c9640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f809c9640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f7c0b53e0 auth_method 0 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-15T13:10:47.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f809c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f809c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.624+0000 7f4f809c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.625+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.625+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.625+0000 7f4f7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.627+0000 7f4f809c9640 10 monclient(hunting): handle_auth_done global_id 17681 payload 824 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.627+0000 7f4f809c9640 1 -- 172.21.15.49:0/1602188201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f7c0b53e0 msgr2=0x7f4f7c0abf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.627+0000 7f4f809c9640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f7c0b53e0 0x7f4f7c0abf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:47.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.627+0000 7f4f809c9640 1 -- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f7c13f660 msgr2=0x7f4f7c0ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:47.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.627+0000 7f4f809c9640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f7c13f660 0x7f4f7c0ad9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:47.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.627+0000 7f4f809c9640 10 monclient: _finish_hunting 0 2024-05-15T13:10:47.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.627+0000 7f4f809c9640 1 monclient: found mon.noname-c 2024-05-15T13:10:47.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.627+0000 7f4f809c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-15T13:10:47.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.627+0000 7f4f809c9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.49:3301/0 2024-05-15T13:10:47.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.627+0000 7f4f809c9640 1 -- 172.21.15.49:0/1602188201 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0adf00 con 0x7f4f7c142870 2024-05-15T13:10:47.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.630+0000 7f4f809c9640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c142870 0x7f4f7c134160 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400c750 tx=0x7f4f7400cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:47.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.630+0000 7f4f78ff9640 10 client.?.objecter ms_handle_connect 0x7f4f7c142870 2024-05-15T13:10:47.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.630+0000 7f4f78ff9640 10 client.?.objecter resend_mon_ops 2024-05-15T13:10:47.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 1 -- 172.21.15.49:0/1602188201 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74017070 con 0x7f4f7c142870 2024-05-15T13:10:47.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:47.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-15T13:10:47.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 10 monclient: dump: 2024-05-15T13:10:47.627 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:47.627 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:47.627 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:47.627 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:47.627 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:47.627 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:47.627 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:47.627 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:47.627 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 10 monclient: _finish_auth 0 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 10 monclient: _check_auth_tickets 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 1 -- 172.21.15.49:0/1602188201 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f74005d40 con 0x7f4f7c142870 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f8242c640 5 monclient: authenticate success, global_id 17681 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f8242c640 10 monclient: _renew_subs 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f8242c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f8242c640 1 -- 172.21.15.49:0/1602188201 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c0acd00 con 0x7f4f7c142870 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 10 monclient: handle_config config(0 keys) v1 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 1 -- 172.21.15.49:0/1602188201 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7400a880 con 0x7f4f7c142870 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f78ff9640 10 monclient: dump: 2024-05-15T13:10:47.628 INFO:teuthology.orchestra.run.smithi049.stderr:epoch 1 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:fsid 8874b84e-6b11-44d3-b531-be645feab39a 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:last_changed 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:created 2024-05-15T12:30:59.910572+0000 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:min_mon_release 19 (squid) 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:election_strategy: 1 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f8242c640 10 client.17681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f8242c640 10 monclient: _renew_subs 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f8242c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f7a7fc640 4 set_mon_vals no callback set 2024-05-15T13:10:47.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f8242c640 1 -- 172.21.15.49:0/1602188201 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c0ad5a0 con 0x7f4f7c142870 2024-05-15T13:10:47.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.633+0000 7f4f8242c640 1 librados: init done 2024-05-15T13:10:47.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.634+0000 7f4f8242c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:47.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.634+0000 7f4f8242c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:47.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.634+0000 7f4f8242c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:47.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.634+0000 7f4f8242c640 1 -- 172.21.15.49:0/1602188201 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7c04f0e0 con 0x7f4f7c142870 2024-05-15T13:10:47.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.655+0000 7f4f78ff9640 1 -- 172.21.15.49:0/1602188201 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4f7401a070 con 0x7f4f7c142870 2024-05-15T13:10:47.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.655+0000 7f4f78ff9640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4f6806ceb0 0x7f4f6806f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-15T13:10:47.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.655+0000 7f4f7bfff640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4f6806ceb0 0x7f4f6806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-15T13:10:47.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.655+0000 7f4f7bfff640 10 monclient: get_auth_request con 0x7f4f6806ceb0 auth_method 0 2024-05-15T13:10:47.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.656+0000 7f4f7bfff640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4f6806ceb0 0x7f4f6806f370 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c036000 tx=0x7f4f6c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-15T13:10:47.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.656+0000 7f4f78ff9640 10 client.17681.objecter ms_handle_connect 0x7f4f6806ceb0 2024-05-15T13:10:47.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.923+0000 7f4f78ff9640 1 -- 172.21.15.49:0/1602188201 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(1251..1251 src has 936..1251) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f4f740563f0 con 0x7f4f7c142870 2024-05-15T13:10:47.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.923+0000 7f4f78ff9640 10 client.17681.objecter ms_dispatch 0x7f4f7c000b90 osd_map(1251..1251 src has 936..1251) v4 2024-05-15T13:10:47.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.923+0000 7f4f78ff9640 3 client.17681.objecter handle_osd_map got epochs [1251,1251] > 0 2024-05-15T13:10:47.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.923+0000 7f4f78ff9640 3 client.17681.objecter handle_osd_map decoding full epoch 1251 2024-05-15T13:10:47.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.923+0000 7f4f78ff9640 20 client.17681.objecter dump_active .. 0 homeless 2024-05-15T13:10:47.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.923+0000 7f4f78ff9640 1 -- 172.21.15.49:0/1602188201 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4f74021020 con 0x7f4f7c142870 2024-05-15T13:10:47.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.923+0000 7f4f78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-15T13:10:47.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:47.923+0000 7f4f78ff9640 10 monclient: _finish_command 1 = system:0 2024-05-15T13:10:47.986 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:17.191 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:47.986 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:17.193 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:47.999 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:17.207 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:48.033 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:17.200 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:48.034 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:17.201 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:48.038 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:17.246 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:48.039 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:17.248 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:48.045 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:17.212 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:48.053 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:17.262 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:10:48.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.129+0000 7f4f8242c640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-15T13:10:48.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.129+0000 7f4f8242c640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-15T13:10:48.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.129+0000 7f4f8242c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.49:3301/0 2024-05-15T13:10:48.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.129+0000 7f4f8242c640 1 -- 172.21.15.49:0/1602188201 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f4f7c0683f0 con 0x7f4f7c142870 2024-05-15T13:10:48.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.420+0000 7f4f78ff9640 1 -- 172.21.15.49:0/1602188201 <== mon.2 v2:172.21.15.49:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4f74058350 con 0x7f4f7c142870 2024-05-15T13:10:48.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.420+0000 7f4f78ff9640 10 monclient: handle_config config(1 keys) v1 2024-05-15T13:10:48.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.420+0000 7f4f7a7fc640 4 set_mon_vals no callback set 2024-05-15T13:10:48.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.420+0000 7f4f7a7fc640 10 set_mon_vals mon_health_to_clog = false 2024-05-15T13:10:48.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.427+0000 7f4f78ff9640 1 -- 172.21.15.49:0/1602188201 <== mon.2 v2:172.21.15.49:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) v1 ==== 123+0+0 (secure 0 0 0) 0x7f4f7c0683f0 con 0x7f4f7c142870 2024-05-15T13:10:48.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.427+0000 7f4f78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-15T13:10:48.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.427+0000 7f4f78ff9640 10 monclient: _finish_command 2 = system:0 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 10 librados: watch_flush enter 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 10 librados: watch_flush exit 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 20 client.17681.objecter shutdown clearing up homeless session... 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 10 client.17681.objecter successfully canceled tick 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 -- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4f6806ceb0 msgr2=0x7f4f6806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4f6806ceb0 0x7f4f6806f370 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c036000 tx=0x7f4f6c03d040 comp rx=0 tx=0).stop 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 10 monclient: shutdown 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 -- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c142870 msgr2=0x7f4f7c134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c142870 0x7f4f7c134160 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400c750 tx=0x7f4f7400cc10 comp rx=0 tx=0).stop 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 -- 172.21.15.49:0/1602188201 shutdown_connections 2024-05-15T13:10:48.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f7c142870 0x7f4f7c134160 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:48.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f7c13f660 0x7f4f7c0ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:48.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.49:6800/3444323000,v1:172.21.15.49:6801/3444323000] conn(0x7f4f6806ceb0 0x7f4f6806f370 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:48.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 --2- 172.21.15.49:0/1602188201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f7c0b53e0 0x7f4f7c0abf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-15T13:10:48.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 -- 172.21.15.49:0/1602188201 >> 172.21.15.49:0/1602188201 conn(0x7f4f7c066ed0 msgr2=0x7f4f7c1420d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-15T13:10:48.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 -- 172.21.15.49:0/1602188201 shutdown_connections 2024-05-15T13:10:48.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 -- 172.21.15.49:0/1602188201 wait complete. 2024-05-15T13:10:48.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-05-15T13:10:48.429+0000 7f4f8242c640 1 librados: shutdown 2024-05-15T13:10:48.433 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-15T13:10:48.433 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-15T13:10:48.433 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-15T13:10:48.433 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:10:48.449 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T13:10:48.447+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 34554) UID: 0 2024-05-15T13:10:48.452 INFO:tasks.ceph.osd.0.smithi049.stderr:2024-05-15T13:10:48.448+0000 b485640 -1 osd.0 1251 *** Got signal Terminated *** 2024-05-15T13:11:02.759 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:31.966 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:02.812 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:31.979 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:02.814 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:32.022 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:02.815 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:31.982 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:02.834 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:32.000 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:02.836 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:32.044 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:02.880 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:32.087 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:02.887 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:32.094 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:02.913 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:32.121 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:03.520 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:32.687 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:03.521 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:32.688 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:03.529 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:32.695 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:03.633 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:32.841 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:03.634 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:32.842 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:03.643 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:32.851 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:03.685 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:32.893 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:03.686 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:32.894 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:03.695 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:32.903 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:22.811 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:52.019 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:22.865 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:52.031 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:22.867 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:52.074 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:22.868 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:52.034 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:22.886 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:52.052 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:22.890 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:52.098 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:22.946 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:52.153 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:22.948 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:52.155 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:22.973 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:52.180 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:23.571 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:52.736 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:23.571 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:52.738 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:23.579 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:39:52.745 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:23.706 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:52.913 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:23.707 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:52.914 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:23.715 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:39:52.923 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:23.726 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:52.934 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:23.727 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:52.935 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:23.737 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:39:52.945 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:42.914 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:12.121 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:42.916 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:12.123 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:42.938 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:12.145 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:42.969 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:12.135 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:42.971 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:12.136 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:42.986 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:12.151 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:42.990 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:12.197 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:42.995 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:12.202 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:43.011 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:12.219 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:43.669 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:12.835 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:43.670 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:12.836 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:43.678 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:12.843 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:43.758 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:12.965 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:43.759 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:12.966 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:43.770 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:12.977 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:43.793 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:13.001 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:43.795 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:13.002 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:43.803 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:13.010 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:11:49.448 INFO:tasks.ceph.osd.0:Stopped 2024-05-15T13:11:49.449 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-15T13:11:49.449 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:11:49.462 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T13:11:49.460+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 34559) UID: 0 2024-05-15T13:11:49.464 INFO:tasks.ceph.osd.1.smithi049.stderr:2024-05-15T13:11:49.462+0000 b485640 -1 osd.1 1255 *** Got signal Terminated *** 2024-05-15T13:11:49.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-05-15T13:11:55.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-05-15T13:12:02.987 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:32.194 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:02.989 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:32.196 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:03.006 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:32.213 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:03.039 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:32.204 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:03.039 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:32.246 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:03.042 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:32.208 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:03.043 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:32.249 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:03.061 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:32.226 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:03.064 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:32.270 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:03.961 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:33.126 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:03.962 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:33.127 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:03.970 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:33.136 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.202 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:33.409 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.210 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:33.417 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.223 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:33.430 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.306 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:33.513 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.307 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:33.514 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:04.316 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:33.523 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:19.955 INFO:tasks.ceph.osd.1:Stopped 2024-05-15T13:12:19.955 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-15T13:12:19.956 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:12:19.969 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T13:12:19.968+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 34574) UID: 0 2024-05-15T13:12:19.969 INFO:tasks.ceph.osd.2.smithi049.stderr:2024-05-15T13:12:19.969+0000 b485640 -1 osd.2 1257 *** Got signal Terminated *** 2024-05-15T13:12:20.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-05-15T13:12:21.685 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:50.891 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:21.692 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:50.898 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:21.718 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:50.925 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:21.745 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:50.910 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:21.746 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:50.911 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:21.751 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:50.958 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:21.755 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:50.962 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:21.778 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:50.943 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:21.798 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:51.004 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.000 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:52.206 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.053 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:52.260 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.064 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:52.271 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.103 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:52.309 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.105 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:52.270 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.107 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:52.313 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.107 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:52.272 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.115 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:52.280 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.116 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:52.322 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.700 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:52.864 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.700 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:52.865 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.708 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:40:52.873 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.729 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:52.935 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.730 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:52.936 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.739 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:40:52.945 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.752 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:52.959 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.755 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:52.962 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:23.767 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:40:52.973 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:26.057 INFO:tasks.ceph.osd.2:Stopped 2024-05-15T13:12:26.057 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-15T13:12:26.057 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:12:26.074 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T13:12:26.069+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 34576) UID: 0 2024-05-15T13:12:26.074 INFO:tasks.ceph.osd.3.smithi049.stderr:2024-05-15T13:12:26.070+0000 b485640 -1 osd.3 1259 *** Got signal Terminated *** 2024-05-15T13:12:26.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-05-15T13:12:26.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-05-15T13:12:32.159 INFO:tasks.ceph.osd.3:Stopped 2024-05-15T13:12:32.159 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-15T13:12:32.159 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:12:32.173 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-15T13:12:32.171+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 34944) UID: 0 2024-05-15T13:12:32.173 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-15T13:12:32.173+0000 b485640 -1 osd.4 1261 *** Got signal Terminated *** 2024-05-15T13:12:32.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-05-15T13:12:32.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-05-15T13:12:38.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-05-15T13:12:38.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-05-15T13:12:38.260 INFO:tasks.ceph.osd.4:Stopped 2024-05-15T13:12:38.261 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-15T13:12:38.261 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:12:38.320 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T13:12:38.314+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 34945) UID: 0 2024-05-15T13:12:38.320 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-15T13:12:38.315+0000 b485640 -1 osd.5 1265 *** Got signal Terminated *** 2024-05-15T13:12:43.126 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:12.332 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.150 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:12.356 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.174 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:12.380 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.175 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:12.340 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.177 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:12.342 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.179 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:12.385 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.182 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:12.388 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.202 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:12.366 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.206 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:12.412 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.850 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:13.015 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.851 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:13.016 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.859 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:13.023 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.874 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:13.080 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.875 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:13.081 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.883 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:13.089 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.897 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:13.103 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.898 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:13.104 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:43.907 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:13.113 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:12:44.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-05-15T13:13:03.190 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:32.396 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.192 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:32.398 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.210 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:32.415 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.238 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:32.402 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.241 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:32.405 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.248 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:32.454 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.249 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:32.455 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.258 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:32.422 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.267 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:32.473 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.972 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:33.136 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.974 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:33.139 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:03.984 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:33.148 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:04.041 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:33.246 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:04.042 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:33.248 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:04.051 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:33.257 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:04.066 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:33.272 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:04.067 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:33.273 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:04.076 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:33.282 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:14.869 INFO:tasks.ceph.osd.5:Stopped 2024-05-15T13:13:14.870 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-15T13:13:14.870 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:13:14.883 INFO:tasks.ceph.osd.6.smithi088.stderr:2024-05-15T13:13:14.880+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 34950) UID: 0 2024-05-15T13:13:14.883 INFO:tasks.ceph.osd.6.smithi088.stderr:2024-05-15T13:13:14.882+0000 b485640 -1 osd.6 1267 *** Got signal Terminated *** 2024-05-15T13:13:20.972 INFO:tasks.ceph.osd.6:Stopped 2024-05-15T13:13:20.972 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-15T13:13:20.972 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:13:20.989 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T13:13:20.983+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 34951) UID: 0 2024-05-15T13:13:20.989 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-15T13:13:20.985+0000 b485640 -1 osd.7 1267 *** Got signal Terminated *** 2024-05-15T13:13:23.191 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:52.396 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:23.246 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:52.451 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:23.265 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:52.470 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:23.292 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:52.456 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:23.295 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:52.459 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:23.300 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:52.506 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:23.303 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:52.509 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:23.311 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:52.475 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:23.322 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:52.527 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:24.011 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:53.175 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:24.012 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:53.176 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:24.021 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:41:53.185 34411== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:24.073 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:53.278 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:24.074 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:53.279 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:24.083 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:53.289 34203== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:24.094 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:53.300 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:24.096 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:53.301 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:24.109 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:41:53.314 34209== Warning: unimplemented fcntl command: 1036 2024-05-15T13:13:24.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-05-15T13:13:27.074 INFO:tasks.ceph.osd.7:Stopped 2024-05-15T13:13:27.074 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-15T13:13:27.075 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-15T13:13:27.075 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:13:27.076 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T13:13:27.083+0000 7f83ea0e9640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 34216) UID: 0 2024-05-15T13:13:27.076 INFO:tasks.ceph.mgr.y.smithi049.stderr:2024-05-15T13:13:27.083+0000 7f83ea0e9640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-15T13:13:27.106 INFO:tasks.ceph.mgr.y:Stopped 2024-05-15T13:13:27.107 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-15T13:13:27.107 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:13:27.135 INFO:tasks.ceph.mgr.x:Stopped 2024-05-15T13:13:27.135 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-15T13:13:27.136 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-15T13:13:27.136 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:13:27.164 INFO:tasks.ceph.mon.a.smithi049.stderr:2024-05-15T13:13:27.159+0000 9f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34177) UID: 0 2024-05-15T13:13:27.164 INFO:tasks.ceph.mon.a.smithi049.stderr:2024-05-15T13:13:27.160+0000 9f99640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-15T13:13:27.908 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:57.113 34203== 2024-05-15T13:13:27.908 INFO:tasks.ceph.mon.a.smithi049.stderr:==00:00:41:57.113 34203== Exit program on first error (--exit-on-first-error=yes) 2024-05-15T13:13:27.933 INFO:tasks.ceph.mon.a.smithi049.stderr:daemon-helper: command failed with exit status 42 2024-05-15T13:13:29.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-05-15T13:13:29.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-05-15T13:13:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-05-15T13:13:33.237 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-05-15T13:13:33.238 INFO:tasks.ceph.mon.a:Stopped 2024-05-15T13:13:33.238 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-15T13:13:33.238 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:13:33.255 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-05-15T13:13:33.250+0000 9f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34179) UID: 0 2024-05-15T13:13:33.255 INFO:tasks.ceph.mon.c.smithi049.stderr:2024-05-15T13:13:33.251+0000 9f99640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-05-15T13:13:33.906 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:42:03.112 34209== 2024-05-15T13:13:33.907 INFO:tasks.ceph.mon.c.smithi049.stderr:==00:00:42:03.112 34209== Exit program on first error (--exit-on-first-error=yes) 2024-05-15T13:13:33.928 INFO:tasks.ceph.mon.c.smithi049.stderr:daemon-helper: command failed with exit status 42 2024-05-15T13:13:34.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-05-15T13:13:34.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-05-15T13:13:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-05-15T13:13:39.340 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-05-15T13:13:39.340 INFO:tasks.ceph.mon.c:Stopped 2024-05-15T13:13:39.341 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-15T13:13:39.341 INFO:teuthology.orchestra.run:waiting for 300 2024-05-15T13:13:39.357 INFO:tasks.ceph.mon.b.smithi088.stderr:2024-05-15T13:13:39.353+0000 9f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34406) UID: 0 2024-05-15T13:13:39.357 INFO:tasks.ceph.mon.b.smithi088.stderr:2024-05-15T13:13:39.354+0000 9f99640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-05-15T13:13:39.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-05-15T13:13:40.023 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:42:09.186 34411== 2024-05-15T13:13:40.023 INFO:tasks.ceph.mon.b.smithi088.stderr:==00:00:42:09.186 34411== Exit program on first error (--exit-on-first-error=yes) 2024-05-15T13:13:40.047 INFO:tasks.ceph.mon.b.smithi088.stderr:daemon-helper: command failed with exit status 42 2024-05-15T13:13:44.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-05-15T13:13:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-05-15T13:13:45.443 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-05-15T13:13:45.443 INFO:tasks.ceph.mon.b:Stopped 2024-05-15T13:13:45.443 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-15T13:13:45.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-05-15T13:13:45.491 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T13:13:45.491 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-15T13:13:45.614 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T13:13:45.615 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-15T13:13:45.673 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T13:13:45.673 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-15T13:13:45.731 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T13:13:45.731 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-15T13:13:45.797 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T13:13:45.797 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-15T13:13:45.913 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T13:13:45.913 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-15T13:13:45.978 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T13:13:45.979 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-15T13:13:46.043 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T13:13:46.044 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-15T13:13:46.101 INFO:tasks.ceph:Archiving mon data... 2024-05-15T13:13:46.103 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091/data/mon.a.tgz 2024-05-15T13:13:46.103 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-05-15T13:13:46.120 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.oZCHkmUk3u 2024-05-15T13:13:46.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.oZCHkmUk3u 2024-05-15T13:13:46.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.oZCHkmUk3u 2024-05-15T13:13:46.408 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.oZCHkmUk3u is 639KB 2024-05-15T13:13:46.526 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.oZCHkmUk3u 2024-05-15T13:13:46.544 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091/data/mon.c.tgz 2024-05-15T13:13:46.544 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-05-15T13:13:46.598 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.xIrGE2jemc 2024-05-15T13:13:46.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.xIrGE2jemc 2024-05-15T13:13:46.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.xIrGE2jemc 2024-05-15T13:13:46.867 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.xIrGE2jemc is 640KB 2024-05-15T13:13:46.970 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.xIrGE2jemc 2024-05-15T13:13:46.986 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091/data/mon.b.tgz 2024-05-15T13:13:46.986 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-05-15T13:13:47.006 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.naPSYYKeTP 2024-05-15T13:13:47.006 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.naPSYYKeTP 2024-05-15T13:13:47.210 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.naPSYYKeTP 2024-05-15T13:13:47.293 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.naPSYYKeTP is 640KB 2024-05-15T13:13:47.399 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.naPSYYKeTP 2024-05-15T13:13:47.414 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-15T13:13:47.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-15T13:13:47.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-15T13:13:47.780 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-15T13:13:47.969 INFO:tasks.ceph:Archiving crash dumps... 2024-05-15T13:13:47.971 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/crash to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091/remote/smithi049/crash 2024-05-15T13:13:47.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-15T13:13:47.999 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091/remote/smithi088/crash 2024-05-15T13:13:48.000 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-15T13:13:48.025 INFO:tasks.ceph:Compressing logs... 2024-05-15T13:13:48.025 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T13:13:48.045 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T13:13:48.068 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41189.log 2024-05-15T13:13:48.068 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40041.log 2024-05-15T13:13:48.068 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40411.log 2024-05-15T13:13:48.068 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.41189.log: /var/log/ceph/ceph-client.admin.40041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63405.log 2024-05-15T13:13:48.078 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63404.log 2024-05-15T13:13:48.078 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34627.log 2024-05-15T13:13:48.078 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.63405.log: /var/log/ceph/ceph-client.admin.63404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40671.log 2024-05-15T13:13:48.078 INFO:teuthology.orchestra.run.smithi049.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40041.log.gz 2024-05-15T13:13:48.078 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107976.log 2024-05-15T13:13:48.078 INFO:teuthology.orchestra.run.smithi049.stderr:gzip/var/log/ceph/ceph-client.admin.40671.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.107388.log 2024-05-15T13:13:48.078 INFO:teuthology.orchestra.run.smithi049.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34627.log.gz 2024-05-15T13:13:48.078 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107976.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38126.log 2024-05-15T13:13:48.078 INFO:teuthology.orchestra.run.smithi049.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40671.log.gz 2024-05-15T13:13:48.078 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107388.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107976.log.gz 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34750.log 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr:gzip/var/log/ceph/ceph-client.admin.38126.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41079.log 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107388.log.gz 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34750.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.63405.log.gz 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41021.log 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38126.log.gz 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.63404.log.gz 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34750.log.gz/var/log/ceph/ceph-client.admin.41079.log: 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40013.log 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.41021.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107682.log 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40411.log.gz 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38507.log 2024-05-15T13:13:48.079 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108186.log 2024-05-15T13:13:48.080 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40625.log 2024-05-15T13:13:48.080 INFO:teuthology.orchestra.run.smithi049.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107682.log.gz 2024-05-15T13:13:48.080 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.108186.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38507.log.gz 2024-05-15T13:13:48.080 INFO:teuthology.orchestra.run.smithi049.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42313.log 2024-05-15T13:13:48.080 INFO:teuthology.orchestra.run.smithi049.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108186.log.gz 2024-05-15T13:13:48.080 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40571.log 2024-05-15T13:13:48.080 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42313.log: /var/log/ceph/ceph-client.admin.39003.log 2024-05-15T13:13:48.085 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40571.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.42313.log.gz 2024-05-15T13:13:48.085 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-15T13:13:48.085 INFO:teuthology.orchestra.run.smithi049.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40013.log.gz 2024-05-15T13:13:48.085 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38728.log 2024-05-15T13:13:48.085 INFO:teuthology.orchestra.run.smithi049.stderr: 87.6%/var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.39003.log.gz 2024-05-15T13:13:48.086 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-15T13:13:48.086 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38728.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38728.log.gz 2024-05-15T13:13:48.086 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38722.log 2024-05-15T13:13:48.088 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.audit.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40571.log.gz 2024-05-15T13:13:48.088 INFO:teuthology.orchestra.run.smithi049.stderr: 92.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36487.log 2024-05-15T13:13:48.088 INFO:teuthology.orchestra.run.smithi049.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-05-15T13:13:48.088 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107430.log 2024-05-15T13:13:48.091 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38722.log: /var/log/ceph/ceph-client.admin.36487.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38722.log.gz 2024-05-15T13:13:48.091 INFO:teuthology.orchestra.run.smithi049.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36487.log.gz 2024-05-15T13:13:48.091 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2024-05-15T13:13:48.091 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39303.log 2024-05-15T13:13:48.091 INFO:teuthology.orchestra.run.smithi049.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107430.log.gz 2024-05-15T13:13:48.091 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40066.log: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.41021.log.gz 2024-05-15T13:13:48.105 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-15T13:13:48.105 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34655.log 2024-05-15T13:13:48.105 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-15T13:13:48.106 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.34655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34855.log 2024-05-15T13:13:48.106 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.audit.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34655.log.gz 2024-05-15T13:13:48.106 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34815.log 2024-05-15T13:13:48.106 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-15T13:13:48.106 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.34855.log: gzip/var/log/ceph/ceph-client.admin.34815.log: -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-15T13:13:48.106 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-15T13:13:48.107 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34855.log.gz 2024-05-15T13:13:48.107 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34815.log.gz 2024-05-15T13:13:48.107 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-15T13:13:48.107 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-05-15T13:13:48.107 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-05-15T13:13:48.107 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-05-15T13:13:48.108 INFO:teuthology.orchestra.run.smithi088.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-05-15T13:13:48.108 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/valgrind/osd.6.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-05-15T13:13:48.108 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-05-15T13:13:48.108 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 64.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-05-15T13:13:48.108 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-05-15T13:13:48.108 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-15T13:13:48.108 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/valgrind/osd.7.log: /var/log/ceph/valgrind/osd.5.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-05-15T13:13:48.109 INFO:teuthology.orchestra.run.smithi088.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-05-15T13:13:48.109 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34695.log 2024-05-15T13:13:48.109 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34775.log 2024-05-15T13:13:48.109 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34695.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34695.log.gz 2024-05-15T13:13:48.109 INFO:teuthology.orchestra.run.smithi088.stderr: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-15T13:13:48.109 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-15T13:13:48.110 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34735.log 2024-05-15T13:13:48.110 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.34775.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-15T13:13:48.110 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.34735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34615.log 2024-05-15T13:13:48.110 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34775.log.gz 2024-05-15T13:13:48.110 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34735.log.gz 2024-05-15T13:13:48.121 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34564.log 2024-05-15T13:13:48.121 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.34615.log: /var/log/ceph/ceph-client.admin.34564.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34615.log.gz 2024-05-15T13:13:48.122 INFO:teuthology.orchestra.run.smithi088.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34564.log.gz 2024-05-15T13:13:48.123 INFO:teuthology.orchestra.run.smithi088.stderr: 94.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-15T13:13:48.140 INFO:teuthology.orchestra.run.smithi049.stderr: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-15T13:13:48.140 INFO:teuthology.orchestra.run.smithi049.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.41079.log.gz 2024-05-15T13:13:48.140 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-15T13:13:48.140 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39303.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28368.log 2024-05-15T13:13:48.140 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.3.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39303.log.gz 2024-05-15T13:13:48.140 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39323.log 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.tmp-client.admin.28368.log: gzip 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28368.log.gz 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38717.log 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42212.log 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38717.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39323.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63401.log 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38717.log.gz 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40277.log 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.42212.log: /var/log/ceph/ceph-client.admin.63401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39970.log 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40277.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.63401.log.gz 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63402.log 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39970.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.42212.log.gz 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107640.log 2024-05-15T13:13:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.63402.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.63402.log.gz 2024-05-15T13:13:48.142 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38980.log 2024-05-15T13:13:48.142 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107640.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40277.log.gz 2024-05-15T13:13:48.142 INFO:teuthology.orchestra.run.smithi049.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107640.log.gz 2024-05-15T13:13:48.142 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 2024-05-15T13:13:48.142 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42211.log 2024-05-15T13:13:48.142 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40921.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38980.log.gz 2024-05-15T13:13:48.142 INFO:teuthology.orchestra.run.smithi049.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.41189.log.gz 2024-05-15T13:13:48.142 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38989.log 2024-05-15T13:13:48.142 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.42211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107766.log 2024-05-15T13:13:48.143 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38989.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38989.log.gz 2024-05-15T13:13:48.143 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51927.log 2024-05-15T13:13:48.143 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107766.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.42211.log.gz 2024-05-15T13:13:48.144 INFO:teuthology.orchestra.run.smithi049.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107766.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38465.log 2024-05-15T13:13:48.144 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:13:48.144 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.51927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39415.log 2024-05-15T13:13:48.144 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38465.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38465.log.gz 2024-05-15T13:13:48.144 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-15T13:13:48.144 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39415.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39415.log.gz 2024-05-15T13:13:48.144 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107556.log 2024-05-15T13:13:48.144 INFO:teuthology.orchestra.run.smithi049.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.51927.log.gz 2024-05-15T13:13:48.144 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.1.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2024-05-15T13:13:48.145 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-15T13:13:48.145 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51926.log 2024-05-15T13:13:48.145 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mgr.y.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107556.log.gz 2024-05-15T13:13:48.145 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39046.log 2024-05-15T13:13:48.145 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.51926.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.51926.log.gz 2024-05-15T13:13:48.145 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40327.log 2024-05-15T13:13:48.145 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39046.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39046.log.gz 2024-05-15T13:13:48.145 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107808.log 2024-05-15T13:13:48.145 INFO:teuthology.orchestra.run.smithi049.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.40625.log.gz/var/log/ceph/ceph-client.admin.40327.log: 2024-05-15T13:13:48.147 INFO:teuthology.orchestra.run.smithi049.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40327.log.gz 2024-05-15T13:13:48.147 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40343.log 2024-05-15T13:13:48.147 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107808.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-15T13:13:48.147 INFO:teuthology.orchestra.run.smithi049.stderr: 87.9%/var/log/ceph/ceph-client.admin.40343.log: -- replaced with /var/log/ceph/ceph-client.admin.107808.log.gz 2024-05-15T13:13:48.147 INFO:teuthology.orchestra.run.smithi049.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2024-05-15T13:13:48.149 INFO:teuthology.orchestra.run.smithi049.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40343.log.gz 2024-05-15T13:13:48.154 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34596.log 2024-05-15T13:13:48.155 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38731.log 2024-05-15T13:13:48.155 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72973.log 2024-05-15T13:13:48.155 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38731.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34596.log.gz 2024-05-15T13:13:48.155 INFO:teuthology.orchestra.run.smithi049.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38731.log.gz 2024-05-15T13:13:48.155 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39029.log 2024-05-15T13:13:48.155 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38273.log 2024-05-15T13:13:48.156 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.72973.log: /var/log/ceph/ceph-client.admin.39029.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39029.log.gz 2024-05-15T13:13:48.159 INFO:teuthology.orchestra.run.smithi049.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.72973.log.gz 2024-05-15T13:13:48.159 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108102.log 2024-05-15T13:13:48.159 INFO:teuthology.orchestra.run.smithi049.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.39970.log.gz 2024-05-15T13:13:48.159 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38721.log 2024-05-15T13:13:48.159 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38273.log: /var/log/ceph/ceph-client.admin.108102.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-15T13:13:48.159 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38721.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38273.log.gz 2024-05-15T13:13:48.159 INFO:teuthology.orchestra.run.smithi049.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108102.log.gz 2024-05-15T13:13:48.160 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-05-15T13:13:48.160 INFO:teuthology.orchestra.run.smithi049.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38721.log.gz 2024-05-15T13:13:48.182 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-05-15T13:13:48.183 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-05-15T13:13:48.183 INFO:teuthology.orchestra.run.smithi049.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-05-15T13:13:48.183 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-05-15T13:13:48.183 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/valgrind/osd.3.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-05-15T13:13:48.183 INFO:teuthology.orchestra.run.smithi049.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-05-15T13:13:48.183 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-05-15T13:13:48.183 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/valgrind/osd.1.log: gzip 64.0% -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-05-15T13:13:48.184 INFO:teuthology.orchestra.run.smithi049.stderr: -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-05-15T13:13:48.184 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42441.log 2024-05-15T13:13:48.184 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/valgrind/mon.c.log: /var/log/ceph/valgrind/osd.2.log: 69.6% 64.0% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-05-15T13:13:48.184 INFO:teuthology.orchestra.run.smithi049.stderr: -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-05-15T13:13:48.184 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41184.log 2024-05-15T13:13:48.184 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.42441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107304.log 2024-05-15T13:13:48.185 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.41184.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.42441.log.gz 2024-05-15T13:13:48.185 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38315.log 2024-05-15T13:13:48.185 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107304.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107304.log.gz 2024-05-15T13:13:48.185 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-15T13:13:48.185 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38315.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38315.log.gz 2024-05-15T13:13:48.185 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39435.log 2024-05-15T13:13:48.188 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mon.a.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41184.log.gz 2024-05-15T13:13:48.188 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39361.log 2024-05-15T13:13:48.189 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39435.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39435.log.gz 2024-05-15T13:13:48.190 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38371.log 2024-05-15T13:13:48.190 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39361.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39361.log.gz 2024-05-15T13:13:48.191 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40436.log 2024-05-15T13:13:48.192 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38371.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38371.log.gz 2024-05-15T13:13:48.193 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35588.log 2024-05-15T13:13:48.199 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40436.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40436.log.gz 2024-05-15T13:13:48.199 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107850.log 2024-05-15T13:13:48.200 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.35588.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35588.log.gz 2024-05-15T13:13:48.201 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107514.log 2024-05-15T13:13:48.201 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107850.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.107850.log.gz 2024-05-15T13:13:48.202 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51925.log 2024-05-15T13:13:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107514.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.107514.log.gz 2024-05-15T13:13:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40964.log 2024-05-15T13:13:48.207 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.51925.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.51925.log.gz 2024-05-15T13:13:48.207 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41133.log 2024-05-15T13:13:48.222 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38730.log 2024-05-15T13:13:48.224 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.41133.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.41133.log.gz 2024-05-15T13:13:48.225 INFO:teuthology.orchestra.run.smithi049.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.40964.log.gz 2024-05-15T13:13:48.225 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39047.log 2024-05-15T13:13:48.226 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38730.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38730.log.gz 2024-05-15T13:13:48.227 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40512.log 2024-05-15T13:13:48.228 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39047.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39047.log.gz 2024-05-15T13:13:48.228 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40176.log 2024-05-15T13:13:48.236 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40512.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40512.log.gz 2024-05-15T13:13:48.237 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107598.log 2024-05-15T13:13:48.250 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40649.log 2024-05-15T13:13:48.251 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107598.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107598.log.gz 2024-05-15T13:13:48.264 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42440.log 2024-05-15T13:13:48.265 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40649.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40649.log.gz 2024-05-15T13:13:48.276 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39465.log 2024-05-15T13:13:48.283 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.42440.log: 91.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-05-15T13:13:48.287 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39465.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39465.log.gz 2024-05-15T13:13:48.291 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40933.log 2024-05-15T13:13:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40790.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42440.log.gz 2024-05-15T13:13:48.304 INFO:teuthology.orchestra.run.smithi049.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2024-05-15T13:13:48.309 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39263.log 2024-05-15T13:13:48.312 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40933.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.40933.log.gz 2024-05-15T13:13:48.326 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107346.log 2024-05-15T13:13:48.326 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39263.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39263.log.gz 2024-05-15T13:13:48.334 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40710.log 2024-05-15T13:13:48.345 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107346.log: 94.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38723.log 2024-05-15T13:13:48.349 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40710.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107346.log.gz 2024-05-15T13:13:48.349 INFO:teuthology.orchestra.run.smithi049.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.40710.log.gz 2024-05-15T13:13:48.349 INFO:teuthology.orchestra.run.smithi049.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40176.log.gz 2024-05-15T13:13:48.349 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-05-15T13:13:48.350 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38723.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38723.log.gz 2024-05-15T13:13:48.350 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72975.log 2024-05-15T13:13:48.362 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.41149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42442.log 2024-05-15T13:13:48.364 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.72975.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.72975.log.gz 2024-05-15T13:13:48.371 INFO:teuthology.orchestra.run.smithi049.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-05-15T13:13:48.371 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108226.log 2024-05-15T13:13:48.375 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.42442.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.42442.log.gz 2024-05-15T13:13:48.375 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40707.log 2024-05-15T13:13:48.376 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.108226.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108226.log.gz 2024-05-15T13:13:48.376 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40081.log 2024-05-15T13:13:48.380 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40707.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40707.log.gz 2024-05-15T13:13:48.380 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38724.log 2024-05-15T13:13:48.389 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40081.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40081.log.gz 2024-05-15T13:13:48.390 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40250.log 2024-05-15T13:13:48.390 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38724.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38724.log.gz 2024-05-15T13:13:48.390 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42314.log 2024-05-15T13:13:48.400 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40136.log 2024-05-15T13:13:48.401 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.42314.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.42314.log.gz 2024-05-15T13:13:48.411 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42210.log 2024-05-15T13:13:48.433 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107934.log 2024-05-15T13:13:48.438 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.42210.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.42210.log.gz 2024-05-15T13:13:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41177.log 2024-05-15T13:13:48.450 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107934.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.107934.log.gz 2024-05-15T13:13:48.458 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39655.log 2024-05-15T13:13:48.468 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.41177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38999.log 2024-05-15T13:13:48.473 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39655.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.41177.log.gz 2024-05-15T13:13:48.476 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39557.log 2024-05-15T13:13:48.480 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38999.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39655.log.gz 2024-05-15T13:13:48.485 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40464.log 2024-05-15T13:13:48.490 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39557.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38999.log.gz 2024-05-15T13:13:48.497 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39599.log 2024-05-15T13:13:48.499 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40464.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39557.log.gz 2024-05-15T13:13:48.519 INFO:teuthology.orchestra.run.smithi049.stderr: 91.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38415.log 2024-05-15T13:13:48.519 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39599.log: -- replaced with /var/log/ceph/ceph-client.admin.40464.log.gz 2024-05-15T13:13:48.519 INFO:teuthology.orchestra.run.smithi049.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39599.log.gz 2024-05-15T13:13:48.527 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41002.log 2024-05-15T13:13:48.542 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38415.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38415.log.gz 2024-05-15T13:13:48.542 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37841.log 2024-05-15T13:13:48.551 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.41002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72974.log 2024-05-15T13:13:48.560 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.37841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40689.log 2024-05-15T13:13:48.563 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.72974.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37841.log.gz 2024-05-15T13:13:48.574 INFO:teuthology.orchestra.run.smithi049.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.72974.log.gz 2024-05-15T13:13:48.587 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40266.log 2024-05-15T13:13:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40689.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.40689.log.gz 2024-05-15T13:13:48.597 INFO:teuthology.orchestra.run.smithi049.stderr: 91.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107892.log 2024-05-15T13:13:48.602 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40266.log: -- replaced with /var/log/ceph/ceph-client.admin.41002.log.gz 2024-05-15T13:13:48.604 INFO:teuthology.orchestra.run.smithi049.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40266.log.gz 2024-05-15T13:13:48.604 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108018.log 2024-05-15T13:13:48.605 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107892.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107892.log.gz 2024-05-15T13:13:48.605 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108060.log 2024-05-15T13:13:48.605 INFO:teuthology.orchestra.run.smithi049.stderr: 92.3%/var/log/ceph/ceph-client.admin.108018.log: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-15T13:13:48.606 INFO:teuthology.orchestra.run.smithi049.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108018.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41138.log 2024-05-15T13:13:48.607 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:13:48.607 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.108060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40030.log 2024-05-15T13:13:48.608 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.41138.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108060.log.gz 2024-05-15T13:13:48.608 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2024-05-15T13:13:48.618 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40030.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.40030.log.gz 2024-05-15T13:13:48.618 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38549.log 2024-05-15T13:13:48.619 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40935.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2024-05-15T13:13:48.619 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39991.log 2024-05-15T13:13:48.621 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.38549.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38549.log.gz 2024-05-15T13:13:48.621 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40506.log 2024-05-15T13:13:48.642 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40946.log 2024-05-15T13:13:48.648 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107472.log 2024-05-15T13:13:48.651 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40946.log: 89.4% 92.3% -- replaced with /var/log/ceph/ceph-client.admin.40506.log.gz 2024-05-15T13:13:48.652 INFO:teuthology.orchestra.run.smithi049.stderr: 96.5% -- replaced with /var/log/ceph/ceph-client.admin.41138.log.gz 2024-05-15T13:13:48.652 INFO:teuthology.orchestra.run.smithi049.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40946.log.gz 2024-05-15T13:13:48.653 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40890.log 2024-05-15T13:13:48.655 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107472.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107472.log.gz 2024-05-15T13:13:48.655 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2024-05-15T13:13:48.657 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40890.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40890.log.gz 2024-05-15T13:13:48.658 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107724.log 2024-05-15T13:13:48.662 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40789.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-05-15T13:13:48.662 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40111.log 2024-05-15T13:13:48.663 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.107724.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107724.log.gz 2024-05-15T13:13:48.664 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34219.log 2024-05-15T13:13:48.671 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40111.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.40111.log.gz 2024-05-15T13:13:48.671 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108144.log 2024-05-15T13:13:48.673 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34219.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.34219.log.gz 2024-05-15T13:13:48.673 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40573.log 2024-05-15T13:13:48.674 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.108144.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108144.log.gz 2024-05-15T13:13:48.674 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63403.log 2024-05-15T13:13:48.677 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.40573.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40573.log.gz 2024-05-15T13:13:48.677 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39080.log 2024-05-15T13:13:48.681 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.63403.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.63403.log.gz 2024-05-15T13:13:48.682 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.39080.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39080.log.gz 2024-05-15T13:13:48.718 INFO:teuthology.orchestra.run.smithi049.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.39991.log.gz 2024-05-15T13:13:49.007 INFO:teuthology.orchestra.run.smithi049.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.40250.log.gz 2024-05-15T13:13:49.608 INFO:teuthology.orchestra.run.smithi049.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40136.log.gz 2024-05-15T13:13:49.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-05-15T13:13:49.698 INFO:teuthology.orchestra.run.smithi088.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-15T13:13:50.345 INFO:teuthology.orchestra.run.smithi049.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-15T13:13:51.749 INFO:teuthology.orchestra.run.smithi049.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-15T13:13:54.969 INFO:teuthology.orchestra.run.smithi088.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-15T13:13:55.050 INFO:teuthology.orchestra.run.smithi088.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-15T13:13:55.493 INFO:teuthology.orchestra.run.smithi088.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-15T13:13:55.506 INFO:teuthology.orchestra.run.smithi049.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-15T13:13:55.814 INFO:teuthology.orchestra.run.smithi049.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-15T13:13:56.185 INFO:teuthology.orchestra.run.smithi088.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-15T13:13:56.187 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-15T13:13:56.187 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m8.105s 2024-05-15T13:13:56.187 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m29.889s 2024-05-15T13:13:56.187 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m1.052s 2024-05-15T13:13:56.755 INFO:teuthology.orchestra.run.smithi049.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-15T13:13:56.766 INFO:teuthology.orchestra.run.smithi049.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-15T13:13:56.768 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-05-15T13:13:56.768 INFO:teuthology.orchestra.run.smithi049.stderr:real 0m8.711s 2024-05-15T13:13:56.768 INFO:teuthology.orchestra.run.smithi049.stderr:user 0m39.473s 2024-05-15T13:13:56.768 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m1.350s 2024-05-15T13:13:56.769 INFO:tasks.ceph:Archiving logs... 2024-05-15T13:13:56.769 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091/remote/smithi049/log 2024-05-15T13:13:56.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T13:14:02.060 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091/remote/smithi088/log 2024-05-15T13:14:02.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T13:14:06.695 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-15T13:14:06.697 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-15T13:14:06.705 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-15T13:14:06.706 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-15T13:14:06.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T13:14:06.709 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-15T13:14:06.803 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-15T13:14:06.812 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-15T13:14:06.812 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T13:14:06.815 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T13:14:06.830 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-05-15T13:14:06.832 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-05-15T13:14:06.834 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T13:14:06.834 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-05-15T13:14:06.834 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv01.front.sepia.ceph.com 3 7 377 69 +1352us[+1592us] +/- 39ms 2024-05-15T13:14:06.834 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -1174us[-1174us] +/- 59ms 2024-05-15T13:14:06.834 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 -751us[ -751us] +/- 81ms 2024-05-15T13:14:06.834 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T13:14:06.836 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T13:14:06.836 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-05-15T13:14:06.836 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 +1302us[+1308us] +/- 40ms 2024-05-15T13:14:06.836 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 72 -1155us[ -881us] +/- 58ms 2024-05-15T13:14:06.836 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -737us[ -731us] +/- 81ms 2024-05-15T13:14:06.837 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T13:14:06.837 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-15T13:14:06.846 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-15T13:14:06.846 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-15T13:14:06.859 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T13:14:06.945 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T13:14:07.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T13:14:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:14:07.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T13:14:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:14:07.074 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-15T13:14:07.083 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-15T13:14:07.097 INFO:teuthology.task.internal:Duration was 3641.473599 seconds 2024-05-15T13:14:07.098 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-15T13:14:07.111 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-15T13:14:07.111 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T13:14:07.114 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T13:14:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T13:14:07.154 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T13:14:07.551 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-15T13:14:07.551 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2024-05-15T13:14:07.551 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T13:14:07.614 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-05-15T13:14:07.614 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T13:14:07.637 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-15T13:14:07.638 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T13:14:07.657 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T13:14:07.705 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-15T13:14:07.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T13:14:07.727 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T13:14:07.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-15T13:14:07.858 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-15T13:14:07.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T13:14:07.885 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T13:14:07.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-15T13:14:07.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:14:07.927 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:14:07.949 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2024-05-15T13:14:07.973 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-05-15T13:14:08.000 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:14:08.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:14:08.022 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T13:14:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T13:14:08.055 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-15T13:14:08.065 INFO:teuthology.task.internal:Transferring archived files... 2024-05-15T13:14:08.065 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091/remote/smithi049 2024-05-15T13:14:08.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T13:14:08.123 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091/remote/smithi088 2024-05-15T13:14:08.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T13:14:08.156 INFO:teuthology.task.internal:Removing archive directory... 2024-05-15T13:14:08.156 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T13:14:08.158 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T13:14:08.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-15T13:14:08.219 INFO:teuthology.task.internal:Not uploading archives. 2024-05-15T13:14:08.219 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-15T13:14:08.335 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-15T13:14:08.335 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T13:14:08.337 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T13:14:08.351 INFO:teuthology.orchestra.run.smithi049.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 13:14 /home/ubuntu/cephtest 2024-05-15T13:14:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 13:14 /home/ubuntu/cephtest 2024-05-15T13:14:08.354 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-15T13:14:08.364 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-15T13:14:08.377 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-15T13:14:08.377 INFO:teuthology.nuke:Checking targets against current locks 2024-05-15T13:14:08.400 DEBUG:teuthology.nuke:shortname: smithi049 2024-05-15T13:14:08.401 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T13:14:08.422 DEBUG:teuthology.nuke:shortname: smithi088 2024-05-15T13:14:08.423 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T13:14:08.426 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:03:26.027077', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEmLW9QW9ln5PsrW3X0qHTFkDw20rYYIOHorXizGP+J22ldFT2fYmnZS5JhVJgCj+5mKoAw8uR2ftY5DxXiwXI='} 2024-05-15T13:14:08.448 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707091', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:03:26.025543', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDHUNiwRS4lgXULAUrwjuTa4tocxkAsPXngfwENURgnL63jMvpSce2NjqdKwO3nce9ZCa2Zb7XDQ1qUVJKThM/Q='} 2024-05-15T13:14:08.470 INFO:teuthology.orchestra.console.smithi049:Power off 2024-05-15T13:14:08.470 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T13:14:08.494 DEBUG:teuthology.orchestra.console.smithi049:power off output: Chassis Power Control: Down/Off 2024-05-15T13:14:08.494 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:14:08.510 INFO:teuthology.orchestra.console.smithi088:Power off 2024-05-15T13:14:08.510 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-15T13:14:08.534 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2024-05-15T13:14:08.534 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:14:08.548 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-05-15T13:14:08.559 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-05-15T13:14:12.550 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:14:12.560 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:14:12.661 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-05-15T13:14:12.670 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-05-15T13:14:16.664 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:14:16.672 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:14:16.774 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2024-05-15T13:14:16.783 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-05-15T13:14:16.876 INFO:teuthology.orchestra.console.smithi049:Power off completed 2024-05-15T13:14:20.785 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-15T13:14:20.895 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2024-05-15T13:14:20.996 INFO:teuthology.orchestra.console.smithi088:Power off completed 2024-05-15T13:14:21.101 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 3641.473598718643 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_rzarzynski@teuthology status: fail success: false 2024-05-15T13:14:21.101 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T13:14:21.199 INFO:teuthology.run:FAIL